Evaluation of Health TechnologiesA Systematic Mapping Study

Priyanka Vegad
Student ID: 13107224
Supervised by: Dr. Pdraig O'Leary

Submitted to the University of Limerick, August 2014 in the partial fulfillment of

the requirements for the Degree of Masters of Science in Software Engineering.

Abstract
Introduction: Objective is to provide an overview of the evaluation techniques used for
assessing health technologies.
Methods: We have applied a systematic mapping study focusing on the evaluation of health
technologies.
Results: Several frameworks were identified in the literature. This research was not restricted to
a particular domain and included Telemedicine, Connected health, Information systems, Health
informatics, e-Health, m-Health, etc. As a result of this mapping study, it was found that there is
little evidence of work done in the evaluation of connected health technologies. Also, there is a
need to focus on technical evaluation of systems and developing a framework which is accepted
universally for the evaluation of any particular area of health technologies.
The research question is: How are Health Technologies assessed?
This research question is broken into the following four sub-questions:
1) What are the frameworks for assessing health technologies?
2) What are the challenges in evaluating health technologies?
3) What are the main evaluation domains?
4) What are the different types of research methods used for evaluation?
Information technology companies and healthcare teams are uniting in the goal of giving their
patients the best service and setting up an improved connection between health care technology
and the patient(s). Technology used in healthcare enables the patient to get updates regarding
their health and allows them to manage their health. But, there are some issues which if ignored
might have a negative effect on the users of the technology. Therefore, there is a need to evaluate
health technologies and the challenges involved in evaluating them.
Conclusions: In order to evaluate Health technologies, the assessment of information technology
is a must. During Information systems evaluation, not only the technology should be assessed but
also the way in which the technology is used by the users/ stakeholders should be taken into
consideration. The emergence of new areas in this field such as connected health is significant
and underlines the importance of adaptive evaluation frameworks.

Declaration

Title: Evaluation of Health Technologies- A Systematic Mapping Study

I hereby declare that this thesis is entirely my own work and does not contain material previously
published by any other author, except where due reference or acknowledgement has been made.
Furthermore, I declare that it has not previously been submitted for any other academic award.

_________________
Priyanka Vegad
August 2014

Acknowledgements

I would like to sincerely thank my supervisor Dr. Pdraig O'Leary for his guidance, enthusiastic
encouragement and valuable suggestions during the planning and development of this research
work. His willingness to give his time so generously has been very much appreciated.
I would also like to thank my parents, Mr. Hitesh Vegad and Mrs. Heena Vegad and my
grandfather, Mr. Mohan Lal Devji Vegad for their love, support and encouragement throughout
to keep me going. I would also like to thank my sister, Payal Vegad for her encouragement and
support throughout to help me complete the research and the dissertation.
Finally, I would like to thank all the lecturers from the CSIS department for all their support
throughout the Masters course.

List of Figures
Figure 1 Research Structure

Figure 2 D&M IS Success Model (Delone and McLean, 1992)

Figure 3 D&M IS Success Model (Delone and McLean, 2003)

10

Figure 4 IT- Organizational Fit Model (Yusof, 2006)

11

Figure 5 Human-Organization-Technology Fit (HOT-fit) Framework (Yusof et al., 2006) 12

Figure 6 TAM Model (Davis et al., 1989)

13

Figure 7 Systematic Mapping Process (Petersen et al., 2008)

17

Figure 8 Research Questions

18

Figure 9 Search Process workflow (da Silva et al., 2011)

20

Figure 10 Phases of screening process

23

Figure 11 Research Domains

30

Figure 12 Research Domain Vs Study type

32

Figure 13: Research Domain Vs Research method

32

Figure 14: Research Domain Vs Year of Publication

38

List of Tables
Table 1 Search String

19

Table 2 List of digital databases searched

22

Table 3 Research Type Facet (Petersen et al., 2008)

24

Table 4 Data Extraction

26

Table of Contents
Introduction

1.1 Introduction

1.2 Research Objective

1.3 Research Structure

Literature Review

2.1 Introduction

2.2 Connected Health

2.3 Health Information Systems

2.4 Evaluating clinical Outcomes of Health Information Systems

2.5 Information System Success
2.6 Clinical Success

13

Research Method

16

3.1 Protocol Development

17

3.1.1 Definition of research Questions

3.1.2 Conduct the search

19

3.1.3 Selection Criteria

20

3.1.3.1 Inclusion criteria

21

3.1.3.2 Exclusion criteria

3.1.4 Screening of papers

21
21

3.1.5 Classification Scheme

3.1.6 Data Extraction
Results

24

24

26

Discussion

36

5.1 Frameworks

36

5.2 Research Domains

37

5.3 Research Methods

39

5.4 Challenges

39

5.5 Limitation of this Research

40

17

Conclusion and Future Works

6.1 Conclusion

41

Key findings

41

6.2 Future works

42

References

43

Appendix A

45

Appendix B

45

41

Chapter 1
Introduction

1.1 Introduction
Information technology companies and healthcare teams are uniting to give their patients the best
of service and to set up an improved connection between health care technology and the
patient(s). Technology is considered as the backbone of health technologies. Innovations in the
field of technology has not only changed the face of healthcare but also has helped to bring some
of the major transformations in the healthcare industry. Innovations include both hardware as
well as software. Hardware innovations include some of the wearable medical devices such as
sensors, and software innovations include electronic personal health record systems such as
Google health and Microsoft Health Vault (Bushko, 2009).
Health technologies had been studied for effectiveness, safety, cost, and various other concerns
(Goodman, 2004). Health technologies includes terms such as e-Health, m-Health, Tele-health,
Digital Health, Telemedicine, Home based care, etc. The fusion of healthcare and technology and
its impact has been a topic of discussion for decades. Research is done and literature related to it
exists. Some of the studies and reports by Leonard and Winkelman 2002; Lorenzi and Riley
1995; Lorenzi and Riley 2000; Lorenzi et al. 1997; Stiell et al. 2003, have focused on identifying
ways to ensure information technology (IT) implementation (Leonard, 2004).
As defined by (Goodman, 2004) Health technology assessment is the systematic evaluation of
properties, effects or other impacts of health technology. Assessment of health technology
includes various properties like safety, economic attributes or impacts, effectiveness, social
impacts, ethical impacts, legal impacts, technical properties, etc. (Goodman, 2004).

1.2 Research Objective

The objectives of this research are listed below:

Identify the gaps in literature to be filled by the researchers in future related to evaluation
of Health Technologies.

Review the studies which are conducted till date (no specific year range) to get a better
understanding of the domain.

Deliver some suggestions for future works in the area of health technologies.

1.3 Research Structure

A Structure for this thesis is described in the figure below:

Figure 1.Research Structure

Chapter 2 includes the literature review of the evaluation of health technologies like Connected
Health, telemedicine, etc. It also includes Information Systems success which is considered
necessary for evaluating health technologies.
Chapter 3 presents the Research method which consists of several phases like Definition of the
research question, conducting a search, selection criteria, screening of papers, the classification
scheme and Data Extraction. This chapter also introduces the research questions and the
screening process followed for this mapping study.
Chapter 4 presents the results for the main research question including the answers to the subresearch questions.
Chapter 5 presents the discussion of the results section in detail. Discussion about the identified
gaps is presented in this chapter.
Chapter 6 presents the conclusion for this mapping study and provides some recommendations
for the future works related to the evaluation of Health Technologies.

Chapter 2
Literature Review
2.1 Introduction
Evaluating Information system success is one of the important steps in evaluating health
technologies. Existing research gives an insight into several frameworks and models for
evaluating Information Systems (IS) Success to name a few: Information Systems Success
Model ( Delone, 2003) and HOT-fit framework (Human, Organization and Technology- fit) (
Yusof et al., 2006).
Previous research and the ongoing research suggests that the purpose of the system used in an
organization can only be entirely successful in delivering its services only when all the
participants using the system are satisfied and are aware of the purpose for which the system is
developed. Information Technology is a key enabler to improving healthcare processes due to its
potential of providing rapid and comprehensive access to information at the point of care (Lenz
and Kuhn, 2004). However, there are some concerns related to the use of technology in
healthcare, which if ignored can have an adverse effect on the users of the technology. Priority is
to provide an overview of the evaluation techniques used for assessing health technologies. With
different interrelated dimensions and variables which will be the base for assessing the quality, it
is necessary to find a common approach to success measurement. In order to determine the
different set of dimensions for assessing the health technologies, review of the already existing
models like Delone and McLean Model of IS Success Model, Human- OrganizationTechnology- fit (HOT- fit) Model, TAM model was done. The different attributes and its
respective dimensions were analyzed and the limitation of each of the models was noted.
Reviewing the existing literature for evaluating health technologies has highlighted some of the
issues which should not be neglected. Some of the frameworks like TEAM framework which
was developed for the evaluation of Health Information system is considered as a global
framework and is too broad for carrying out the IS evaluation. The HOT-IT fit organizational

model proposed by (Yusof et al., 2006) needs to be tested in clinical settings in order to validate
its usefulness.

2.2 Connected Health

Connected Health encompasses terms such as wireless, digital, electronic, mobile, and
telehealth and refers to a conceptual model for health management where devices, services or
interventions are designed around the patients needs, and health related data is shared, in such a
way that the patient can receive care in the most proactive and efficient manner possible. All
stakeholders in the process are connected by means of timely sharing and presentation of
accurate and pertinent information regarding patient status through smarter use of data, devices,
communication platforms and people (Caulfield and Donnelly, 2013).
Advancements in the technologies used in the past few decades has helped evolve healthcare
enabling remarkable advances in connected health (Bushko, 2009). Current healthcare systems
are concerned with only one part of the health continuum for diagnosing and treating patients
(Saranummi, 2013).
The aim of connected health technologies is to deliver health care and monitoring from the
hospital and into the customers homes. A further step in the advancement in the technology will
assist in monitoring the patient irrespective of where the patient is. Software installed on devices
like mobile phones with the help of internet enables continuous monitoring of parameters related
to patients health such as heart rate, ECG, sleep monitoring, etc. by receiving data or signals
(Bushko, 2009). The technology used in healthcare makes the patient aware of their health
related issues. In connected health, individuals are equally responsible and are partners with
healthcare professionals in the involvement for managing their health and wellness (Saranummi,
2013). The primary goal for adopting health technologies is to provide patients the best service

possible by gathering and interpreting accurate information which will help them to take correct
decisions on time which reduces the cost, time and effort thereby resulting in the timely
treatment of the patient.

2.3 Health Information Systems

Health information systems (HIS) is a patient-centered approach that aims at contributing a high
quality; efficient patient cares (Haux, 2006). Health care is beneficial in many ways by the
adoption of Health Information Systems. HIS consists of a collection of different processes
which are implemented to assist and give support to the health care organization to achieve their
objectives with accuracy which can be done by enhancing the efficiency and effectiveness
(Yusof et al., 2006). The barriers related to the use of HIS should also be considered while
evaluating HIS because these obstructs will decide whether the HIS is successful or a failure
depending on the particular context. Two common barriers identified regarding the use of
healthcare systems are culture and process changes. Other barriers include low levels of expertise
existing in the healthcare organization, lack of medical staff, changes in the traditional workflow
patterns (Yusof et al., 2006).
Because of the power of economic forces and community, patient-centered health care
information system is accepted by people these days. This patient-centered health information
system allows providers to keep the patients record updated and easily accessible when needed
(Menp et al., 2009). It is seen that while developing a system (Information system),
continuous change in the conditions or requirements can be expected. Healthcare institutions too
represent an environment of changing requirements. It is not fair to believe that all the
information and knowledge management process in healthcare are systematically mapped to an
IT system. Rather, the physicians, nurses and patients also integrate knowledge and communicate
it with each other, while IT systems complements and support these processes (Lenz and Kuhn,
2004). Building a safer healthcare system is suggested where healthcare professional should
increase their understanding about information technology and how it could be applied to deliver
safer healthcare (Schrenker, 2006). To achieve the expected results, information systems should
perform accordingly. Moreover, to ensure the information system performs correctly, the quality
of the software used should be assessed. This determines whether the information system is
successful or not.
The use of IS can achieve remarkable benefits for an organization by not only gaining
competitive advantage, increasing productivity, automation of operational decision, shorter

product cycle and supporting of strategic and tactical decisions but also helps in enhancing the
overall impact on organizational forms and management paradigms ( Mohd Al-adaileh, 2009).
However, in order to achieve these valuable benefits, success of ISs is a must. Several measures
had been taken to evaluate HIS (Health Information Systems). It is noted that attempts made to
adopt health care technologies that require data entry by the healthcare professionals have not
been entirely successful (Van Der Meijden et al., 2003).

2.4 Evaluating clinical Outcomes of Health Information Systems

Evaluation is performed to improve HIS by learning from the experiences and identifying more
effective ways to improve the systems (Yusof et al., 2006). Factors that affect the evaluation
include organization factors and human factors. Human factors include a range of issues like
training, personnel attitudes, ergonomics (the study of people's efficiency in their working
environment) and regulations affecting employment (Yusof et al., 2006).
For healthcare information system to work appropriately it requires an advanced health
information network, that supports clinical care, personal health management, the reduction of
avoidable mistakes in population health and research, and evidence-based medicine (Menp et
al., 2009). Different approaches to HIS evaluation developed based on a number of factors like
technical, economic, human, organizational, etc. are discussed by (Yusof et al., 2006). Research
works shows that there are a number of frameworks, specially developed for evaluating Health
information systems.
To name a few includes 4Cs framework, CHEATS framework, TEAM framework, ITAM
framework, etc. (Yusof et al., 2006). The 4Cs stands for (a) Communication (Communication
between the departments), (b) Care (Medical care delivery), (c) Control (Control in the
organization and (d) context (Clinical settings). The model developed for the evaluation of IT in
healthcare was CHEATS which includes characteristics like Clinical, Human and Organization,
Administrative, Technical and Social. This model helped in assessing and providing more
definite measures for clinical settings. Total Evaluation and Acceptance methodology (TEAM)
consists of three dimensions i.e. Role, Time and Structure. TEAM is considered as a global
framework and is too broad for carrying out a particular IS evaluation. The IT Adoption Model
(ITAM) primarily focuses on an individual users point of view. ITAM is insufficient when it

comes to evaluating organizational impact. But, they complement each other and help in
evaluating different aspects of HIS (Yusof et al., 2006).
There is a growth in the building of Regional Health Information infrastructure to provide the data
accessibility anytime and anywhere keeping the data secure and safe. Its underlying idea is to
improve the healthcare through quality, completeness and timeliness of public health data. This will
help the timely detection of disease (Menp et al., 2009). Regional Health Information Systems
(RHIS) focuses on five main areas i.e. flow of information, collaboration, process redesign, system
usability and organization culture. Based on their study they highlighted some of the outcomes
which include timely information, clinical data exchange and improved communication between the
professionals (Menp et al., 2009). The reason behind the existence of RHIS was to improve the
quality of care, the health and safety of individuals and improves the effectiveness. RHIS offers
better-patient-centered care which reduces the risk to patients' life. During the late nineties, the
concept of Integrated Electronic health records and patient-centered healthcare integrated with RHIS
has been suggested for a period of more than 30 years (Menp et al., 2009). According to the
authors, the outcomes of Regional Health Information Systems were better collaboration and
communicating resulting in better flow of information, process design, usability, etc. It was noted
that the RHIS was also responsible for enhancing the data access, allowing the clinical data
exchange between the different systems in distinct environments and providing the real-time patient
information (Menp et al., 2009). Because of the presence of diverse systems, the healthcare
industry faces some issues which include lack of integration between the different Health
Information Systems (HISs) used lack of communication also results in making the task difficult for
the healthcare professionals and the absence of a standardized Electronic health records (EHR). The
existing problem of integrating all the patients' data from all the different systems into one and was
not completely efficient in making the data accessible by the healthcare professionals. It was noted
that there was no standardized Electronic Health Record and because of the different EHR
structures, it was difficult to accumulate the vast data records of the patients and use it for treatment
purpose. This problem leads to the duplicity of records, unnecessary treatment, and inaccuracy in
medical records and increase in effort and cost. Maintaining an EHR also allows the patients to enter
their details manually that require the patient to input valid details related to them. Incorrect data
input might result in risk to the patient's information.

This problem of integrating all the patient records into one was overcome with the advancement in
technology. Google Health is a new platform which is an example of an electronic personal health
record system. It allows the patients to store their personal health information in an account that is
accessible anytime and from anywhere. Not only it allows entering the information manually, but
also allows the users (healthcare professionals and patients) to import their medical records from
their hospitals. Another example is the Microsoft HealthVault, which like Google Health integrates
personal health information. This platform needs to set up a free Microsoft account which permits
the patients to gather, store and search health information online. Microsoft Health Vault is not only
confined to personal health record, it acts as an online storage of information allowing applications
to interact with the existing data, analyzing it and providing solutions (Bushko, 2009). Active
Health is an enhanced personal health record which uses the Microsoft health vault to extract
patient's records (Bushko, 2009).

2.5 Information System Success

Initial step in evaluating the success of the information systems is to define what success means.
According to (Van Der Meijden et al., 2003) Determination of success depends on the setting, the
objective, and the stakeholders. Moreover, to check whether the system was successful or not, a
detailed study should be carried out based on a specific setting (Van Der Meijden et al., 2003).

Evaluating the success of Information Systems (ISs) is identified as one of the most crucial
issues in IS field. Information system infuses almost all facet of human life. Because of its
continuous use and presence, high investments are done which demands for success and quality
of the systems (Iivari, 2005). However, in order to achieve these valuable benefits, success of
ISs is a must. To date much research has been conducted to identify the factors that contribute to
Information System Success. Many researchers suggested different aspects of IS Success, which
demands a more integrated view of the concepts of IS Success (DeLone and McLean, 1992).
There is no good in measuring various independent or input variables, if the dependent or output
variable I/S success cannot be measured with a similar degree of accuracy (DeLone and
McLean, 1992).

The model developed by Delone and McLean is one of the most cited models for IS Success.
Ultimately, the different aspects of IS success was drawn together into a single descriptive model
referred to as D&M IS Success Model (DeLone and McLean, 1992) which included six
interdependent variables to measure the success of IS which includes: system quality,
information quality, user satisfaction, systems use, organizational impact and individual impact
(DeLone and McLean, 1992).

Figure 2. D&M IS Success Model (DeLone and McLean, 1992)

According to (Van Der Meijden et al., 2003), the characteristics with which the information
processing system is assessed includes the system quality attributes (accessibility, usability, etc.),
Information quality attributes (accuracy, completeness, legibility), User satisfaction can include
both system itself or its information. According to Delone and McLean, their paper presented an
IS Success model which attempted to capture the interdependent nature of IS Success (Delone,
2003). In their original model, Delone and McLean introduced the model as a casual-explanatory
model of how information quality and system quality affect user satisfaction and use, how use
and user satisfaction are correlated and how they affect each other , how individual impact leads
to organizational impact and how use and user satisfaction are direct antecedents of individual
impact (Iivari, 2005). A huge discussion continues pertaining to the use of the appropriate set of

variables that can be used to measure ISs Success. The original model has been reviewed and
modified by many researchers. Some of them even introduced frameworks for measuring the IS
Success effectively like the framework proposed by Grover et al., used an alternative,
theoretically based perspective to build a theoretically based construct space for IS effectiveness
that compliments and extends the [ Delone and McLean] IS Success Model (Delone, 2003).
However, the success model does not serve as a basis for the selection of suitable IS measures
(Delone, 2003). In 2003, Delone and McLean proposed The Updated D&M IS Success model
which consists of six major categories of I/S success: Service Quality, System Quality, System
Use, Information Quality, User satisfaction and Net Benefits.

Figure 3. Updated D&M IS Success Model (Delone, 2003)

(Seddon and Kiew, 2007) critically examined the meaning of these four constructs and the
support of relationships between the different variables. They considered D&M IS Success
Model as their foundation for their research and listed the difference between Delone and
McLean's model and the model tested. These differences are as follows: (a) One of the variables
Use in Delone and McLeans model was replaced by Usefulness (b) introduction of a new
variable System Importance which according to the authors will help to clarify the variations in
users view of usefulness and user satisfaction (c) the two- way dependency between user and

1
0

User satisfaction in Delone and McLeans model was replaced by a one- way dependency
between usefulness and user satisfaction (Seddon and Kiew, 2007).
Human- Organization- Technology Fit (HOT-fit) Framework
By considering the strengths and weaknesses of IS Success model and the IT-fit Model, and
integrating the best features of both the models into one, the HOT-fit framework is developed.
IT -fit Model
The concept of fit between technical, organizational and social factors is analyzed in order to
identify the gap between the present healthcare systems and features of the new system (Yusof et
al., 2006). IT-fit model includes both internal and external elements and the concept of fit
between the main organizational elements. Internal-fit is achieved by a dynamic equilibrium of
organizational components like business strategy, organizational structure, management
processes, and roles and skills. External fit is attained by considering organizational strategies
based on environment which includes: market, industry and technology (Yusof et al., 2006)

Figure 4. IT- Organizational Fit Model

Human-Organization-Technology Fit model as stated by the author is capable of identifying the

organizational elements which effects the IS Success and also provides a fit between them and
1
1

includes factors like Human (Roles and skills), Organization (Strategy, Structure and
Management process) Technology (IT). This model complements the Delone and McLean Model
of IS Success. The factors of HOT- fit are again categorized into several different dimensions (as
illustrated in figure 5) based on the dimensions described in D&M IS Success Model. These
three factors consists of eight interrelated dimensions of HIS success namely: System Quality,
Information Quality, Service Quality, System Use, User Satisfaction, Organizational Structure,
Organizational Environment and Net Benefits (Yusof et al., 2006).

Figure 5. Human-Organization-Technology Fit (HOT-fit) Framework

(Yusof et al., 2006)
TAM Model
Technology Acceptance Model (TAM) ( Mohd Al-adaileh, 2009) is a model IT adoption and
operation that has been customized to explain computer usage. Figure 6 Illustrate the model
elements such as perceived usefulness, behavioral intention to use, perceived ease of use and the
relationships among them.

1
2

Figure 6. Source Davis et.al. (1989), Venkatesh et.al. (2003)

The above described model emphasizes the importance of perceived usefulness. Author believed
that the assumptions regarding the perceived usefulness might be true based on users level of
proficiency ( Mohd Al-adaileh, 2009).
It is observable that so far there is no agreement on the use of a set of measures of IS Success. It
is not easy to choose some particular variables from the list of variables available. In reassessing
these variables, no single measure is essentially better than another; so to make a choice of a
success variable depends on the objective of the study, the aspect of the information system
which is addressed by the study, the organizational context, the independent variable under
investigation, the level of analysis, i.e. individual, organization, or society and the research
method (DeLone and McLean, 1992).

2.6 Clinical Success

The adoption of new technologies from a wide range of industries and sectors is not new but it is
still considered as a challenge to integrate technology with healthcare

(Leonard, 2004).

Evaluation is considered as an important factor as it serves a number of purposes. Given the

unpredictable characteristics of IS in general and the aim of improving clinical performance and
patient outcomes in particular, evaluation is undertaken to understand system performance
(Yusof et al., 2006). While evaluating the clinical success the clinical evidence is collected,
analyzed and compared. Moreover, this raises some important issues which need to be taken into

13

consideration which includes: what to measure and how to measure (Jen-Hwa Hu, 2003). Many
papers discuss the factors that affect both information technology and systems. However, none of
the studies so far provides a description on how to operationalize (Leonard, 2004). The solution
to this is to summarize a brief list of critical success factors (CSFs) which will provide guidelines
to set the framework for achieving the success in designing, developing and implementing new
technology in healthcare. The challenge faced is to present these listed CSFs within a framework
that is both informative and illustrative (Leonard, 2004).
(Jen-Hwa Hu, 2003) introduced a new model to access the success of the systems used in
telemedicine. For developing this new model, he followed Delone and McLean Model of IS
success as the basis of his new model. This new revised model was designed considering the
various aspects of healthcare and the technologies used in it. A new dimension was added to the
model known as the input data quality. The idea behind adding this dimension for assessing the
success of the system was to focus on the importance of inputting the data onto the system in
connected health settings. The quality of input from connected clinical perspective is crucial
which has a strong impact on the resulting services along with the participants of the system and
the organization. In this model, several feedbacks loops are also integrated which are different
from the original Delone and McLean model.
One of the many solutions which is capable of dealing with significant healthcare problems is the
involvement of the service providers (of primary, secondary and tertiary care) and the patients
who are receiving treatment, the electronic data exchange allows the creation of virtual service
networks which are capable of dealing with significant problems in modern healthcare; e.g.,
service accessibility, quality, utilization effectiveness/efficiency, and costs (Jen-Hwa Hu, 2003).
(Leonard, 2004) Discuss some of the case studies to elaborate on the issues relating to the CSFs
(Critical success factors) of technology adoption. He discussed the acceptance of electronic
images of x-rays. During the conversion process from traditional film- based radiology process
to electronic images of x-rays, doctors were notified that their patients x-ray reports will be
emailed to them. Many doctors complained about not receiving the e-mails, but the case was
they did not check their e-mails. To overcome this problem, the patients reports were sent to the
doctors using computer diskette or compact discs (depending on the size of the data to be sent).
Even after making minor changes in order to help doctors getting acquainted with this

1
4

transformation, doctors complained about the delay in getting the results. Because they found it
difficult to match the arrival of these hardware supplies containing information to their
information needs. This does not mean that the doctors had no experience in identifying the
connection between the Compact discs and patient x-rays. Technology, they just were not
familiar to it. Taking into consideration this finding, the imaging department decided to continue
the use of CDs but in a different way. They finalized on a solution of sending the CDs via
internal hospital delivery system in traditional large x-ray envelopes. This helped the doctors to
recognize the type of packaging in which the CDs were placed in (Leonard, 2004).
Other factors, apart from clinical success that relates to the overall success in health technologies
includes Economic, operational, strategic and technological (Duarte and Costa, 2012). These are
considered as the four dimensions along the information system life cycle. These dimensions are
discussed in brief in the following section (Duarte and Costa, 2012):
Technological Dimension: It involves product, its characteristics and its alignment with the
business strategy. Companies follow this these days to acquire a place in the competitive market.
Operational Dimension: It includes people and the processes. This dimension is believed to be
an important one because it is related to all the people in the organization. Not only this, IS
system involves huge investment by the organizations.
Economic Dimension: It involves the cost and benefits for an investment project.
Strategic Dimension: It involves the business vision and the organizational strategy.

1
5

Chapter 3
Research Method
A systematic literature review (SLR) is a form of secondary study (A study that reviews all the
primary studies which is focused on a definite research question with the aim of
integrating/synthesizing evidence related to a specific research question) which deals with
identifying, evaluating and interpreting all available research with respect to a particular research
question, or phenomenon of interest, or topic area. (Kitchenham and Charters, 2007). Another
type of reviews other than systematic literature review includes systematic mapping studies.
Even before carrying out systematic review focusing on a particular domain, if very little
evidence exists regarding the subject or the chosen topic of research is very broad, in that case a
systematic mapping study might be an appropriate solution (Kitchenham and Charters, 2007).
A mapping study focuses on classifying the relevant literature based on the particular topic and
aggregate studies concerning the defined categories. These categories usually are categorized
depending upon the publication information such as authors name, authors affiliation,
publication source, publication date etc. (Kitchenham et al., 2011).
The reason behind conducting a systematic literature review is that it helps identify gaps in
current research in order to suggest areas which need further investigation. It also helps in
providing a framework or background to suitably position novel research activities (Kitchenham
and Charters, 2007). The early stages of systematic literature review and systematic mapping
study are similar which includes: (a) Identification of primary studies which may contain
significant research results (b) Selection of appropriate primary studies based on the inclusion
and exclusion criteria (c) if required, quality assessment of the selected studies is performed
(Budgen et al., 2008). Two stages where mapping studies differs strongly from systematic
literature review are during the classification of the papers and during data extraction and
aggregation(Kitchenham et al., 2011)

1
6

According to (Kitchenham and Charters, 2007), systematic literature review consists of three
different phases: Planning the Review, Conducting the Review, Reporting the Review. In the
following section, the process used for this mapping study is discussed in detail.

3.1 Protocol Development

We have applied systematic mapping study focusing on the evaluation of health Technologies.
Various process steps for systematic mapping study which includes the definition of research
questions, conducting a search for relevant papers, screening of papers, key wording of abstracts
and data extraction and mapping (Petersen et al., 2008). Each and every step has an outcome
depending on how the subsequent steps are performed. The outcome of this process is the
systematic map.

Figure 7: The systematic Mapping Process (Petersen et al., 2008).

3.1.1 Definition of research Questions

The research question that directs this mapping study is: How are health technologies assessed?
This primary research question was identified for carrying out the mapping study. Sub- questions
were derived from the primary questions which are illustrated in Figure 8.

1
7

Primary Question

How are Health Technologies

assessed?

What are the frameworks

for assessing health
technologies?

What are the challenges

What are the main

in evaluating health
technologies?

evaluation domains?

What are the types of

research methods used
for evaluation?

Figure 8 Research Questions

Q1. What frameworks exist for assessing Health Technologies? This question identifies the
frameworks and models for evaluating information system success. To check whether the system
is successful or not, a detailed study should be carried out based on a specific setting(Van Der
Meijden et al., 2003). Evaluating the success of Information systems (ISs) is identified to be the
most crucial issues in IS field. Information system infuses almost all facet of human life.
Because of its continuous use and presence, high investments are done which demands for
success and quality of the systems (Iivari, 2005).
Q2. What are the challenges in evaluating health Technologies? This question includes the
barriers and the challenges involved in adopting and evaluating health technologies.
Q3. What are the main evaluation domains? Rather than focusing on just one domain, different
domains were considered for this study. Broadening the research domain might result in finding
a more general solution to the problem of the absence of study in a particular domain.
Q4. What are the types of research methods used for evaluation? Studies selected for this
research are categorized on the basis of research method. Studies included for this research

18

includes both qualitative and quantitative which involves questionnaires, interviews, case study
etc.

3.1.2 Conduct the search

The terms selected for conducting the search were defined based on the approaches described by
(Kitchenham et al.), which followed a strategy with the below-mentioned stages:
(1)

Derive major terms from the main research question as well as the sub- questions.

(2)

Identify alternative spellings and synonyms for major terms.

(3)

Checking the keywords.

(4)

Use of Boolean OR operator

(5)

Use of Boolean AND operator to merge the different keywords.

Table 1. Search Strings

connected health OR health information technologies OR HIT OR information technology

OR healthcare OR telemedicine OR telehealth OR digital health OR health information
systems OR hospital information systems OR health services OR health informatics OR
medical informatics OR biomedical informatics OR assisted living, OR e-Health OR
mHealth.
AND
assessment OR evaluation OR success factors OR assessment frameworks OR
challenges OR outcomes OR key performance indicators OR success OR assess
variables OR performance measurement OR systematic review.'

OR and AND operators are used to merge different keywords which make the search process
simple. The search process used for conducting this mapping study was taken from (da Silva et
al., 2011) and is shown in Figure 9. This process mainly consists of three steps: (a) search strings
shown in Table 1 are entered in each of the digital databases (e.g. IEEE Explore, ACM Digital
library etc.). The entire list of the digital databases searched is illustrated in Table 2. (b) Manual

1
9

search was also conducted based on the year of publication, the area, etc. (c)Snow- balling
which means following the references of the papers already found for the mapping study.

Manual Search

Automatic Search

Inclusion/
Exclusion

Snow-Balling

Primary Studies

Figure 9. Search Process workflow (da Silva et al., 2011).

The focus was to search the area of research for the relevant papers. Terms such as telemedicine,
tele-health, digital health, e-Health, etc. were used to find relevant literature based on the health
technologies. Both automatic and manual search were conducted. The set of inclusion and
exclusion criterias made the selection process easier and helped in narrowing down the research
domain which is discussed in the following section:

3.1.3 Selection Criteria

In this section, the relevant papers are searched using the search string (Table 1) based on some
inclusion and exclusion criteria which helped to eliminate articles which were not appropriate for
the study.

20

3.1.3.1 Inclusion criteria

Below mentioned are the inclusion criterions set for this study:
1

The study must investigate the approaches, methods, challenges involved related
to the assessment of the health technologies.

Papers reflecting various works depicting different assessment frameworks/

models are included.

Terms such as telemedicine, telehealth, digital health, eHealth, etc. were

considered.

Peer-reviewed papers, i.e. conference, journal, book chapters, and workshop

proceedings.

Only papers written in English were accepted for the study.

No specific year range was selected for conducting the search.

3.1.3.2 Exclusion criteria

1

Studies that are only available in PowerPoint presentations are excluded.

Studies which only deals with healthcare but lacks IT integration is excluded.

Studies relating to the domain and relating to the research questions were only
included.

3.1.4 Screening of papers

Screening of the papers for the selection of the relevant paper was done in phases. Figure 10
illustrates the different phases of the screening process.
In stage 1, electronic sources were used to search studies related to the topic. List of all the
sources is given in Table 2. No specific year range was selected for this study. A specific number
of papers were selected at the end of the first stage by quickly reviewing the title of the papers
found. Papers were searched automatically by inputting the search string and by manually
inputting the keywords but were irrespective of the year of publication.
In stage 2, Papers that satisfied the inclusion and exclusion criteria are included. Numbers of
papers were found associated with the topic and the research questions. So, n= 1003 papers were
considered for conducting this study at the end of the second stage.

2
1

In stage 3, In order to determine whether the papers were related and answer to the research
questions or not, title and the keywords of the papers were reviewed. Based on the exclusion
criteria, the number of papers selected was narrowed down to n= 89 papers. Primary focus was
to identify the appropriate papers for the study. Many papers related to the integration of IT and
healthcare were obtained as a result of the conducted search but, only the papers discussing
assessment methods and frameworks involved in assessing health technologies and the success
factors associated are included. As a result, out of 1003 papers, 914 papers were excluded.
In Stage 4, abstracts of the papers were read to get a deeper understanding about the content of
the paper. At the end of this stage, papers were chosen for an in-depth study. Results were
extended by going through all references of each article and inspecting them for relevant studies.
This process of reviewing the references of the selected articles was done to make sure that none
of the important papers were left out. At the end of this stage, n= 40 papers were downloaded in
PDF format.
In stage 5, an in-depth study of all the papers from stage 4 was done and the papers that do not
justify the research area were excluded.
List of Digital databases
IEEE Explore
Google scholars
Science Direct
ACM Digital Library
Scopus
Wiley InterScience
Table 2.List of digital databases searched.

2
2

Stage 1
Conduct search using
the search strings

Stage 2
Include the relevant
papers

Library search (n=1003)

Stage 3
Eliminate non-relevant
papers based on the
exclusion criteria by
reading the keywords
and title.
Titles considered
relevant (n=89)
Stage 4
Detail review of the papers
based on the abstract and
extending results

N
o
Stage 5

p
a
p
e
r
s
s
e
l
e
c
t
e
d
a
f
t
e
r

review based on abstracts (n=40)

I
n
d

e
p

23

3.1.5 Classification Scheme

Results of the studies were classified based on (Petersen et al., 2008) idea of categorizing.
Studies were classified by reconsidering the abstracts, titles, keywords and sometimes even
reviewing the introduction and the conclusion. This classification scheme help identify the
different paper type across two different facets i.e. Research Facet and Context Facet. A
description of each research category is presented in Table 3.
Research Facet: Validation Research, Evaluation Research, Solution Proposal, Philosophical
Papers, Opinion Papers, and Experience Papers.
Context Facet: Connected Health, Telemedicine, Information system, Health information
system, e-Health, m-Health, Tele-health, Assisted Living, and Health Informatics.

Table 3 .Research Type Facet (Petersen et al., 2008)

3.1.6 Data Extraction

To address the objectives of this study, data was gathered by reviewing the papers in detail. The
following information was extracted from the papers selected for the study:
1

Authors name

2 publication year

2
4

1 Concept
2

Context Facet

Type of study

Research method (Analysis)

5 Measures
6

Models/frameworks evaluated,

7 Models/ frameworks identified in the study.

Gathering all the data in an excel sheet helped us to put down all the information in one place.
By analyzing the data, categorizing the types of papers, detecting the approach used for the
studies and identifying the frameworks for evaluation of IT in healthcare were represented using
bubble plot, bar chart and pie-chart. These types of representations are used for representing all
the data in a pictorial way that makes it easy for the readers to understand.

2
5

Chapter 4
Results
Main findings of this research are listed in this chapter based on the research questions. This
research was primarily focused on evaluating the connected health technologies but due to the
lack of evidence, the research area was extended to health technologies. The research question
including the sub research question is illustrated in Figure 8.
Connected Health is one of the emerging paradigms in healthcare domain, and this area needs to
be explored carefully in order to get the best out of it. This study not only focuses to the
evaluation of Health Technologies covers domains like telemedicine, e-Health, m-Health, Telehealth, Bio-medical informatics, Assisted living, Health Information systems, etc. It also
emphasis on the challenges involved in evaluating health technologies.
Overviewing the systematic literature in IT healthcare, many models were identified for
evaluating health technologies. Evaluation frameworks are not only confined to one domain but
several which gives an insight on the different parameters on which each domain is evaluated
differently.
4.1 RQ1- What are the frameworks for assessing Health Technologies? Table 4, shows the list of
models that have been evaluated by different authors for their respective studies. Among the
models discussed, Delone and McLean model (both original and updated) are the most popular.
Out of 40 papers analyzed, 7 papers referred to Delone and McLean model for IS Success
evaluation as the basis for their research or evaluated the IS Success model. Different authors
made an effort to add different dimensions and variables to the already existing Delone and
McLean model of IS Success.
Studies also considered other frameworks as a basis of proposing a new framework for
telemedicine evaluation, IS Success etc. After gathering data and considering all the evaluation
models, Delone and McLean model of IS Success (original and updated) (1992 and 2003) was
evaluated by 7 authors.

26

Title
Year
Information systems success: the
quest for the dependent variable.

The DeLone and McLean Model

of Information Systems Success:
A Ten-Year Update

Author
Concept
1992 William H. DeLone, The objective is to present an
Ephraim R. McLean integrated view of the concept of
IS Success and because of which
a taxonomy is introduced in this
paper. The taxonomy presents
some major dimensions and
variables for evaluating the IS
Success. The focus was to bring
awareness and provide structure to
the "dependent variable" of IS
Success.
2003 DeLone, William H In this paper, research contribution
McLean, Ephraim R made regarding IS Success and
the efforts made to update the
original DeLone and McLean
Information Systems (IS) Success
Model is discussed.

Model
IS Success
Model

Evaluated by
William H. DeLone
Ephraim R. McLean-2003,
Paul Jen-Hwa Hu -2003,
Raid Mohd Al-adaileh 2009,
William H. DeLone
Ephraim R. McLean-2002

IS Success
Model- Updated

Raid Mohd Al-adaileh - 2009

Maryati Mohd. Yusof, Ray J.
Paul, Lampros K. Stergioulas 2006,
Ana Isabel Mendes Duarte,
Carlos J. Costa- 2012,
Maryati Mohd. Yusof,
Anastasia Papazafeiropoulou,
Ray J. Paul, Lampros K.
Stergioulas- 2008

An evaluation of information
systems success: A user
perspective-The case of Jordan
telecom.

2009 Mohd Al-adaileh,

Raid

Towards a Framework for

Health Information Systems
Evaluation

2006 Mohd, Maryati

Paul, Ray J
Stergioulas,
Lampros K

This paper focuses on reviewing

the available Information system
success evaluation studies. A new
framework for evaluation was
proposed and the model was
verified based on the research
findings.
This paper combines the
approaches of existing models of
IS Evaluation mostly on two
models IT-Organization Fit and IS
Success Model. The proposed
framework was evaluated and was
suggested that it can be used as a
tool to conduct enhanced HIS
evaluation. A framework for
evaluating Information System in
healthcare setting inorder to ensure
effective implementation and
positive impact of HIS on
healthcare delivery.

A model for evaluation of Mohd Al-adaileh, Raid-2009.

information system
success

HOT-Fit framework

Mohammed, Siti Asma

Yusof, Maryati Mohd-2013

Table 4

(DeLone and McLean, 1992) Proposed a framework for evaluating IS Success with dimensions
System Quality, Information Quality, Use, User satisfaction, Individual Impact, Organizational
impact. This model captured the multidimensional and interdependent nature of IS Success.
Being one of the most famous model, hundreds of research have challenged, validated and
applied the original model discussed and stated the need to find a common approach to success

2
7

measurement (Delone, 2003). Due to the increasing impact of IS on society, and an increase in
the IS research, the authors felt the necessity to review, evaluate, and update the original success
model. Later, this model was self-evaluated and the dimensions modified for this model were
Information Quality, system quality, service quality, use, user satisfaction, net benefits. The
authors of the paper, suggested that intention of use might result into one of the worthwhile
alternatives for the updated Delone and McLean model of IS Success. They even substituted the
measure Use to Intension of use because the former is a behavior and latter being an attitude
(Delone, 2003). Dimensions like Individual Impact and Organizational impact were combined
into one single variable called Net benefits.' Net benefits refers to both the measures i.e. Use
and User satisfaction.'
A revised model of Delone and McLean model of IS Success was proposed by (Jen-Hwa Hu,
2003) in which the measures used for the model differed from the original Delone and McLean
model. They were: Input data quality, system quality, information quality, system use, User
satisfaction, service impacts, Individual impacts, organizational impacts. Comparing both the
models, the new IS Success dimension i.e. Input data quality was included. In the original model,
Delone and McLean only considered the quality of the information provided by the system.
Quality of input data is also one of the important factors in evaluating IS Success because the
resulting services will get effected by the quality of input data (Jen-Hwa Hu, 2003). The revised
model also includes service quality as an additional dimension.
Organizations major concern is the failure of IS Success. To increase the rate of IS success, IS
evaluation should be given importance. ( Mohd Al-adaileh, 2009) Proposed a conceptual model
based on the Delone and McLean model of IS Success (original and updated) and TAM
(Technology acceptance model). This model was used to test an evaluation model of IS Success
from the users perspective. The dimensions used for the proposed model are Information
quality, perceived ease of use, perceived usefulness, user technical capabilities, management
support, user's perception of IS success. Systems ease of use was excluded from this model.
HOT-Fit model (Yusof et al., 2006) of HIS evaluation was developed using Delone and McLean
model for evaluation of IS Success and IT- fit model. This new model complements the Delone
and McLean model and consists of seven dimensions of HIS success: System Quality, Service

2
8

Quality, System Use, Net benefits. Information Quality, Structure, environment and User
satisfaction Authors modified the above-listed frameworks for their studies in order to derive
new frameworks from these existing models. These frameworks also helped as validation of the
various existing models. From this discussion, it is clear that Delone and McLean model of IS
Success and several other models which used this model have some dimensions in common such
as Information quality, system quality, service quality and user satisfaction. This states that these
dimensions were considered necessary for the evaluation purpose. On the other hand, dimensions
like input data quality, individual impact, organizational impact, user technical capabilities, etc.
were added considering the different aspects for which the model is used.
Using the snowballing technique, other relevant frameworks/models were identified. ( Mohd Aladaileh, 2009) TAM (Technology acceptance model) is considered as one of the popular model
of IT adoption. TAM including other models like its evolution TAM2, the Diffusion of
Innovation (DOI), the Unified Theory of Acceptance and Use of Technology (UTAUT), an
Extension of the UT AUT Model and the Delone and McLean Model of Information Systems
Success are developed by the researchers to examine various aspects like acceptance, diffusion,
adoption and success of Information Systems (Srur and Drew, 2012).
TRA (Theory of Reasoned Actions) (Topacan et al., 2008) is considered as one of the models
that explain the human behavior in the adoption process. Later, an additional construct was
added to the model and the new model was named Theory of Planned Behavior (TPB). For
predicting technology acceptance in the medical sector, Daviss TAM model is considered as
more suitable as compared to TRA to predict technology acceptance in the medical sector.
(Glasgow, 2007) discussed two of the evaluation frameworks, one is RE-AIM which stands for
Reach, Effectiveness, Adoption, Implementation or consistency of program delivery, and
Maintenance or sustainability. RE-AIM model focuses on both individual and setting level
factors which are critical
for public health impact
PROCEED model.

(Glasgow, 2007) and the other model discussed is PRECEDE-

Other model discussed in the study are TDQM- Total data quality management, TIQM- total
information quality management, PRISM, The Canadian Institute for Health Information (CIHI)

2
9

IQ framework, AHIMA data quality management model. HOT-fit model proposed by (Mohd et
al., 2006) combines the approaches of existing models of IS Evaluation mostly on two models
IT-Organization Fit and IS Success Model. The proposed framework was evaluated and was
suggested that this framework can be made into use as a tool to conduct enhanced HIS
evaluation. A framework for evaluating Information System in healthcare setting in order to
ensure effective implementation and positive impact of HIS on healthcare delivery.

4.2 RQ2- What are the main evaluation domains? Analyzing the results based on the
classification of publications according to the domains, majority of the studies were based on
telemedicine evaluation. Reviewing the studies selected for this research, studies were grouped
based on the different domains. A pie-chart representation is used to illustrate the distribution of
studies indifferent domains. Figure 11, demonstrates about 40% of the studies are based on
Telemedicine, 22% covers HIS (Health information system) domain, 17% focuses on IS
evaluation. Few studies discuss about the remaining domains. Out of 40 studies considered,
which were searched using the search string (Table 1), only 5% of the studies were related to
connected health evaluation.

Domains
3%

3% 0% 3%

Connected Health
Telemedicine

5%
7%

IS
Health information
systems
40%

22%

e-Health

m-Health
Telehealth
Assisted Living
17%

Health informatics

Figure 11 Research Domains

3
0

A total of 40 papers were used for in-depth analysis. Papers were grouped together based on the
classification scheme discussed in Table 3. Bubble plot representation was used to represent the
different domains and the type of paper identified. Figure 12 presents the two dimensions i.e.
Context Facet, which includes the different domains (e.g. connected health, telemedicine, Health
information system, health informatics etc.) and Research Facet which includes the type of
papers (e.g. Evaluation research, validation research, solution proposal, etc.)
The highest number of papers focuses on the telemedicine domain and IS research and their
research types are Evaluation Research and Philosophical paper respectively. Figure 12 gives a
clear picture that none of the study described an opinion paper. Insignificant number of study
described validation research. The most common type of the research dimension is philosophical
papers and Evaluation Research.
Among the list of selected studies, some were a combination of more than one type of research
facet. Similarly, some studies covered more than one domain in their study e.g. (Le Gldic et al.,
2011) discussed more than one domain i.e. both telemedicine and e-health systems.
Discussing about Connected Health Technologies evaluation, (Mathur et al., 2007) and (O'Neill
et al., 2012) frameworks used for evaluating connected health. Discussed a functional framework
for the analysis of diabetes care improvement programs that used communication technology and
provides a systematic approach to the problem involved was proposed by (Mathur et al., 2007).
The framework used for analysis consisted of 3 stages: 1) Identification of functional
components 2) Identification of structural components, 3) Identification of the level of
automation in program design. Functional components used for this framework includes: Remote
monitoring, dynamic feedback, motivation support, coordination of care, structured Education.

3
1

Figure 12
4.3 RQ3- What are the types of research methods used for evaluation? The types of analysis that
helped us categorize the selected articles for the study are:
1)

Qualitative Analysis

2)

Quantitative analysis

Figure 13, it is evident that about 90% of the studies followed the qualitative approach for
analysis whereas 15% adopted quantitative approach. At times, combinations of both qualitative
and quantitative methods are used. In some case qualitative methods are used to prepare
quantitative studies and sometimes to support qualitative argumentation, quantitative
measurements are used (Ammenwerth et al., 2009). In this study, a small number (10%) used
both the approaches.

Figure 13

32

In order to specify the type of methodology used and classify the studies, different research
methods such as interview, questionnaires, meetings, case study etc. that are adopted by studies
included in this research.
Majority (12%) studies undertook Questionnaire and interviews as part of their research
methodology. The second major research method which were identified in the study was
participant observation or patient interventions which covered 10% of the all the research
methods used in the identified studies. It needs to be noted that some studies did not specify the
type of the survey method used, those studies were included under Survey (7%). Meetings and
case studies were also a part of this evaluation with 3% and 5% respectively.
4.4 RQ4- What are the challenges in evaluating health Technologies? As stated by (Jen-Hwa
Hu,

2003), challenges relating to technology and management should be addressed by

healthcare organizations in order to achieve the ultimate success of telemedicine. During IT

evaluation, not only the technology but also the approach in which the technology is used by the
users should be taken into account (Ammenwerth et al., 2003). Telemedicine evaluation should
not be confined to examine only the factors which lead to the effectiveness of the telemedicine
application but also evaluation should determine why was it not effective (Brear, 2006).
(Ammenwerth et al., 2003) discuss the problems, its consequences and the possible solutions for
the problems in evaluation of IT in healthcare. They are:
1. Complexity of the evaluation object
2. Complexity of the evaluation project
3. Motivation for evaluation
Problems related to the complexity of the evaluation object are while evaluating the information
systems. Evaluation will not only focus on the hardware and software but also on the users
interaction with the system. Not only this, evaluation also deals with other aspects such as the
way the technology or system is introduced and used in the organization, how useful it is for the
users, etc.
Consequences (Ammenwerth et al., 2003) to the problem of complexity of the evaluation
includes 1) IT usability changes during the first stage of evaluation. As a result, a considerable

3
3

amount of time is required to implement the technology and to get the evaluation results. 2) Due
to changes in work organization and staff, the use of information technology might get affected
and change with time. The authors referred to this problem as the moving evaluation target. 3)
Type of users based on their system knowledge, the workflow, and the functionalities of the
information system used in different departments might differ even if the information systems
are similar.
Complexity of the evaluation project is also considered as one of the other problem areas in
evaluating connected health technologies. Evaluation is usually performed in real and complex
health care environment and should consider all kinds of stakeholders of the technology which
includes different professional groups like nurses, doctors, patients, etc. Evaluation methods will
differ depending on the different view point such as economic, sociologic, psychological,
organizational, technical, information logistical, or clinical (Ammenwerth et al., 2003).
Consequences to the problem of complexity of the evaluation project includes: 1) Different
stakeholder has different evaluation questions which make it difficult to form a definite
evaluation criteria. 2) Evaluation question may change depending on the intermediate evaluation
results, changes in stakeholders opinion or the information technology itself. 3) Selection of a
suitable evaluation design or evaluation method is often regarded as a problem during evaluation
studies.
Third problem area includes motivation for evaluation (Ammenwerth et al., 2003). Evaluation is
only possible if the stakeholders agree to take part in this evaluation process. Evaluation study is
conducted only when there sufficient number of participants and funding available.
Consequences to the above-mentioned problem include: 1) Due to the lack of participants
willingness to be a part of the evaluation study, often there is a shortage of getting sufficient
resources. 2) The problem of external validity can be minimized by including the participants for
the study who are happy to volunteer.
Challenges related to evaluation of telemedicine include identifying many techniques and sources for
collection of health information (Brear, 2006). Communication between telemedicine applications
between two sites may differ because of the different ways in which the information

3
4

is recorded and stored. Different sites may have a different context, individual perspective and
different ways of using the technology (Brear, 2006) which makes the evaluation result to differ
from one organization to other.
(Heathfield et al., 1998) discuss the challenges involved in evaluating information technology in
healthcare. According to the author, clinical systems are embedded social systems with different
people, providers, settings, etc. There is a need to recognize and predict the behavior of systems
and deliver essential knowledge to notify further developments. Successful interdisciplinary
working requires overcoming cultural and methodological division which involves setting up of
trust and mutual respect (Pagliari, 2007).

3
5

Chapter 5
Discussion
This research was previously planned to identify the evaluation techniques for connected health
evaluation but because of the unavailability of much literature related to connected health.
Connected health domain is considered as an emerging paradigm and a lot needs to be explored.
We broadened the research area to Health technologies. The results of this systematic mapping
study indicate that there is still little information on the evaluation of connected health
technologies.
Results obtained from this mapping study could be put into use by other researchers as evidence
and can continue this research based on the results obtained.

5.1 Frameworks
Among the list of selected papers, the most referred model/framework for Information system
success assessment is Delone and McLean model of IS Success. Studies used this model for their
research and for developing a new model. Almost all the studies which used IS Success model as
their basis for research initially evaluated the model and then argued on the addition or
elimination of specific dimensions based on the context. To date, there is no particular
framework which is accepted universally for the evaluation of any particular area of health
technologies. The solution to this problem is by developing a model with different dimensions
which can be customized for evaluating the different health technologies. Analyzing all the
selected papers for this study it is noticed that not many evaluation frameworks were actually
evaluated (either self-evaluated or evaluated by others). This study identified only 4 different
frameworks which were evaluated.
Connected Health also involves technology and assessment of information system success is
required. Delone and Mclean model which is used for evaluating the IS Success can be used for
evaluating connected health technologies. Dimensions can be selected from a set of dimensions
by referring various evaluation frameworks.

3
6

Some of the dimensions of the IS Success model for telemedicine evaluation which were
considered important were: Information quality, system quality, service quality and user
satisfaction. Out of 40 studies, 7 studies evaluated the original (1992) and updated (2003) Delone
and McLean IS Success model. For the successful implementation of IT in healthcare, IS
Success is necessary. Because of the popularity of IS Success model by Delone and McLean,
many research were conducted in this domain, the popularity is also seen in the results obtained
from this study. Results of this study show that more focus is on one model i.e. Delone and
McLean model which is because of the popularity of the model but there will be no good when
concentrating on just one evaluation model. Other models related to IS Success and Health
technologies evaluation needs to be evaluated in order to extract new dimensions or variables.

5.2 Research Domains

This research work presented in the form of research work is not only confined to a specific
domain of health technologies rather it includes several other domains like Telemedicine, eHealth, m-Health, Health Information systems, telehealth, Connected health, etc. Because of the
large involvement of Information technology in healthcare, making it more effective also
involves some risks. This makes evaluation of the health technologies a necessity.
Telemedicine being one of the oldest technologies used in healthcare, a good number of papers
could be found related to this domain. A lot of Research has been conducted in this field of
telemedicine in the past few decades and from this study we found that about 40% of the papers
discussed telemedicine. Recent works are related to connected health technologies and a few
papers can be found which shows the progress in research in this domain. Figure 14 shows the
domain and the year in which the different studies were published.

3
7

Figure 14. Research Domain Vs Year of Publication

Primary focus of this mapping study is to identify the various frameworks used for the
assessment of Health technologies. When conducting the search for relevant papers, many
studies focusing on telemedicine assessment methods and approaches were identified.
Data is represented by classifying the papers according to research facets discussed in Table 3
and the context facet which includes different domain of IT in healthcare. It is noted that
considering those 40 selected papers, the most common type of research facet used for the study
is Evaluation Research and Philosophical Paper. Only 2 studies used validation research for their
study which includes techniques which were new and were not yet implemented (Petersen et al.,
2008). Philosophical papers provide a new approach of looking at the already existing models by
organizing the field in the form of a conceptual framework (Petersen et al., 2008). Possibly,
developing a framework/ model right from the beginning is considered difficult as compared to
evaluating already existing models. It is much easier to evaluate and validate the existing model
rather than developing a new model. In this study, none of the papers was opinion papers as these
type of paper expresses opinion of the authors and usually do not depend on earlier works related
to it (Petersen et al., 2008). The reason for selecting evaluation paper type perhaps is that
previous research works related to a specific area gives the new research base or a direction for
further developments/research in the field. Systematic mapping studies could be very useful and

38

can help provide support for the new research because starting a research in a specific domain
which is entirely new requires much time and effort.

5.3 Research Methods

In order to conduct research in any of the domains in healthcare, we need to investigate a wide
range of topics, identify the gaps in the literature which gives an idea about the different issues
and how further research can provide a solution which can be beneficial to society.
Quantitative research method narrows down the result to numbers. Result obtained from
quantitative research is easy for other researchers to start a discussion because it is too direct.
Qualitative research, on the other hand, gives an in-depth understanding on a specific topic and
gives a clear picture of what needs to be done. For the in-depth analysis, qualitative research
method takes time and effort.
Sometimes, both qualitative and quantitative approaches are used for a study. A qualitative
method used along with quantitative research justifies the response for the study. This study
followed both qualitative and quantitative research as we intended to review the literature and
carried out a mapping study based on the research conducted. A huge difference in the use of
research methods can be seen from the results obtained (Figure 12). It was found that among the
list of selected studies, 90% of the studies followed qualitative method for research. Majority of
the papers identified in this study followed qualitative research despite of the fact that it takes
more time, effort and it not suitable for obtaining statistical data.

5.4 Challenges
To increase the rate of successful implementation of the health technologies, challenges should
be taken into consideration. Much research includes challenges in health technologies, but very
few challenges were identified in the evaluation of health technologies.
While there is emerging evidence of the challenges involved in evaluating some new areas of
health technologies like connected health. Evaluation needs to be conducted based on the context
of implementation. Evaluation results may vary depending on the type of participants,
stakeholders, sites, context in which it is implemented, etc.

3
9

5.5 Limitation of this Research

This research might have experienced some limitations. These are discussed below:
1

Not all the studies are included in this research. Some studies might not have
appeared in the search conducted because of the keywords in the search strings and could
be missed.

Scientific libraries selected for the selection of studies related to the evaluation of
health technologies were kept minimum keeping time into account.

To make sure all the major studies related to the topic were included, the snowballing technique was used which took additional time than finding the relevant papers
using the search string.

During data extraction stage, classification of the study was done based on
authors judgement. Regardless of carefully extracting the data, there might be a
possibility of classifying it incorrectly.

4
0

Chapter 6
Conclusion and Future Works
6.1 Conclusion
In this paper we presented a systematic mapping study on the evaluation of health technologies.
This study provides an initial evidence for evaluation in the field IT healthcare.
This research includes a review of the existing frameworks for the evaluation of health
technologies regardless of the healthcare IT domain. Among the frameworks identified in the
study, some were evaluated and others were

theoretical model (yet to be implemented).

Dimensions of different frameworks were also analysed. A comparison of some of the

dimensions of the identified frameworks were done in order to identify the reason for adopting
some dimensions/ variables over others. Health technologies evaluation model can be applied to
several context depending on the requirements.
Inorder to accept a technology in a hospital setting, focus should be on the evaluation of
technology. The success of the piece of technology depends on how well the technology is used
by its user, the context in which it is applied, and how effective the technology will be after
implementing it to hospital settings. Evaluating technology will help reduce the risks.
Assessment of Information systems is a must for evaluating Health technologies. During IT
evaluation, not only the technology but also the approach in which the technology is used by the
users/ stakeholders should be taken into consideration. The emergence of new areas in this field
such as connected health is significant and underlines the importance of adaptive evaluation
frameworks.

Key findings
6.1.1

Frameworks for assessing health technologies:

A good number of frameworks were identified while reviewing the literature. Delone and
Mclean model is evaluated by many researchers for their study which provide a
framework for the evaluation of Information system success.

4
1

6.1.2

Main evaluation Domains:

Highest numbers of studies were identified in telemedicine. It is found that very little
research is done in the area of connected health.

6.1.3

Different types of research methods used for evaluation:

Two types of analysis that helped us categorize the selected articles are quantitative
analysis and qualitative analysis. It is found that Questionnaire and Interviews methods
are used by most studies in this research.

6.1.4

Challenges in evaluating health technologies:

Research works are identified which includes the challenges or barriers of health
technologies but very few studies discuss about the challenges involved in evaluating
health technologies.

6.2 Future works

This research does not focus on any specific domain. Narrowing down the research area and
concentrating in any particular domain might help to find the solution to the problem. Not much
research has been done in Connected Health domain. Future research work is necessary in this
area. A standardized model for health technologies needs to be developed which can be
universally accepted.

42

References
AMMENWERTH, E., GRBER, S., BRKLE, T. & ILLER, C. 2009. Evaluation of health
information systems: Challenges and approaches. Handbook of Research on
Advances in Health Informatics and Electronic Healthcare Applications:
Global Adoption and Impact of Information Communication Technologies.
AMMENWERTH, E., GRBER, S., HERRMANN, G., BRKLE, T. & KNIG, J. 2003.
Evaluation of health information systemsproblems and challenges.
International journal of medical informatics, 71, 125-135.
BREAR, M. 2006. Evaluating telemedicine: lessons and challenges. Health
Information Management Journal, 35, 23.
BUDGEN, D., TURNER, M., BRERETON, P. & KITCHENHAM, B. Using mapping
studies in software engineering. Proceedings of PPIG, 2008. 195-204.
BUSHKO, R. G. 2009. Strategy for the Future of Health, Ios Press.
CAULFIELD, B. M. & DONNELLY, S. C. 2013. What is Connected Health and
why will it change your practice? QJM, hct114.
DA SILVA, I. F., DA MOTA SILVEIRA NETO, P. A., O'LEARY, P., DE ALMEIDA, E. S. & DE
LEMOS MEIRA, S. R. 2011. Agile software product lines: a systematic
mapping study. Software: Practice and Experience, 41, 899-920.
DELONE, W. H. 2003. The DeLone and McLean model of information systems
success: a ten-year update.
Journal of management information systems, 19, 9-30.
DELONE, W. H. & MCLEAN, E. R. 1992. Information systems success: the quest
for the dependent variable. Information systems research, 3, 60-95.
DUARTE, A. I. M. & COSTA, C. J. Information systems: life cycle and success.
Proceedings of the Workshop on Information Systems and Design of
Communication, 2012. ACM, 25-30.
GLASGOW, R. E. 2007. eHealth evaluation and dissemination research. Am J Prev
Med, 32, S119-26. GOODMAN, C. S. 2004. Introduction to health technology
assessment. The Lewin Group. virginia, USA. HAUX, R. 2006. Health information
systemspast, present, future. International journal of medical
informatics, 75, 268-281.
HEATHFIELD, H., PITTY, D. & HANKA, R. 1998. Evaluating information technology in
health care: barriers and challenges. bmj, 316, 1959.
IIVARI, J. 2005. An empirical test of the DeLone-McLean model of information
system success. ACM SIGMIS Database, 36, 8-27.
JEN-HWA HU, P. Evaluating telemedicine systems success: a revised model.
System Sciences, 2003. Proceedings of the 36th Annual Hawaii
International Conference on, 2003. IEEE, 8 pp.
KITCHENHAM, B., MENDES, E. & TRAVASSOS, G. H. A systematic review of cross-vs.
within-company cost estimation studies.
KITCHENHAM, B. A., BUDGEN, D. & PEARL BRERETON, O. 2011. Using mapping
studies as the basis for further researcha participant-observer case study.
Information and Software Technology, 53, 638-651.
KITCHENHAM, B. A. & CHARTERS, S. 2007. Guidelines for performing systematic
literature reviews in software engineering.
LE GLDIC, S., FOURNIER, M., MALEK, S. & DUPRAZ-POISEAU, A. Telemedicine and ehealth systems: Regulatory framework and issues. 2011 the 14th
International Symposium on Wireless Personal Multimedia Communications:
Communications, Networking and Applications for the Internet of Things,
WPMC'11, 2011.

4
3

LENZ, R. & KUHN, K. A. 2004. Towards a continuous evolution and adaptation of

information systems in healthcare. International journal of medical
informatics, 73, 75-89.
LEONARD, K. J. 2004. Critical success factors relating to healthcares adoption of
new technology: a guide to increasing the likelihood of successful
implementation. Electronic Healthcare, 2, 72-81.
MENP, T., SUOMINEN, T., ASIKAINEN, P., MAASS, M. & ROSTILA, I. 2009. The
outcomes of regional healthcare information systems in health care: A
review of the research literature. International Journal of Medical Informatics,
78, 757-771.
MATHUR, A., KVEDAR, J. C. & WATSON, A. J. 2007. Connected health: A new
framework for evaluation of communication technology use in care
improvement strategies for type 2 diabetes. Current Diabetes Reviews, 3,
229-234.
MOHD AL-ADAILEH, R. 2009. An evaluation of information systems success: A user
perspective-The case of Jordan telecom group. European Journal of Scientific
Research, 37, 226-239.
MOHD, M., PAUL, R. J. & STERGIOULAS, L. K. Towards a Framework for Health
Information Systems Evaluation. HICSS, 2006.
O'NEILL, S. A., NUGENT, C. D., DONNELLY, M. P., MCCULLAGH, P. & MCLAUGHLIN, J.
2012. Evaluation of connected health technology. Technology and Health
Care, 20, 151-167.
PAGLIARI, C. 2007. Design and evaluation in eHealth: challenges and implications
for an interdisciplinary field. Journal of medical Internet research, 9.
PETERSEN, K., FELDT, R., MUJTABA, S. & MATTSSON, M. Systematic mapping
studies in software engineering. 12th International Conference on
Evaluation and Assessment in Software Engineering, 2008. 1.
SARANUMMI, N. 2013. In the Spotlight: Health Information Systems.
Biomedical Engineering, IEEE Reviews in, 6, 21-23.
SCHRENKER, R. A. 2006. Systems and software engineering contribute not only to
advancing and improving the delivery of healthcare but also to doing it
more safely than has been the case in the past.
SEDDON, P. & KIEW, M.-Y. 2007. A partial test and development of DeLone and
McLean's model of IS success. Australasian Journal of Information
Systems, 4.
SRUR, B. L. & DREW, S. Challenges in designing a successful e-health system for
Australia. Information Technology in Medicine and Education (ITME), 2012
International Symposium on, 3-5 Aug. 2012 2012. 480-484.
TOPACAN, U., BASOGLU, A. N. & DAIM, T. U. Exploring the success factors of health
information service adoption. Management of Engineering & Technology,
2008. PICMET 2008. Portland International Conference on, 27-31 July 2008
2008. 2453-2461.
VAN DER MEIJDEN, M., TANGE, H. J., TROOST, J. & HASMAN, A. 2003.
Determinants of success of inpatient clinical information systems: a
literature review. Journal of the American Medical Informatics
Association, 10, 235-243.
YUSOF, M. M., PAUL, R. J. & STERGIOULAS, L. K. Towards a framework for health
information systems evaluation. System Sciences, 2006. HICSS'06.
Proceedings of the 39th Annual Hawaii International Conference on, 2006.
IEEE, 95a-95a.

4
4

Appendix A
List of studies selected
1. AHERN, D. K., PATRICK, K., PHALEN, J. M. & NEILEY, J. D. 2006. An introduction to
methodological challenges in the evaluation of eHealth research: Perspectives from the Health eTechnologies Initiative. Evaluation and Program Planning, 29, 386-389.

2. ALI, A., FAISAL, A. & SORWAR, G. Telemedicine framework for Bangladesh. IEEE Region 10 Annual
International Conference, Proceedings/TENCON, 2010. 335-338.

3. AMMENWERTH, E., GRBER, S., HERRMANN, G., BRKLE, T. & KNIG, J. 2003. Evaluation of
health information systemsproblems and challenges. International journal of medical informatics, 71,
125-135.

4. AOKI, N., DUNN, K., JOHNSON-THROOP, K. A. & TURLEY, J. P. 2003. Outcomes and methods in
telemedicine evaluation. Telemedicine Journal and e-Health, 9, 393-401.

5. BASHSHUR, R. L., REARDON, T. G. & SHANNON, G. W. 2000. Telemedicine: a new health care
delivery system. Annual Review of Public Health, 21, 613-637.

6. CLARKE, M. & THIYAGARAJAN, C. A. 2008. A systematic review of technical evaluation in

telemedicine systems. Telemedicine and e-health, 14, 170-183.

7. DELONE, W. H. 2003. The Delone and McLean model of information systems success: a ten-year update.
Journal of management information systems, 19, 9-30.

8. DELONE, W. H. & MCLEAN, E. R. 1992. Information systems success: the quest for the dependent
variable. Information systems research, 3, 60-95.

9. DELONE, W. H. & MCLEAN, E. R. Information systems success revisited. System Sciences, 2002.
HICSS. Proceedings of the 35th Annual Hawaii International Conference on, 2002. IEEE, 2966-2976.

10. DHILLON, H. & FORDUCEY, P. G. Implementation and evaluation of information technology in

telemedicine. System Sciences, 2006. HICSS'06. Proceedings of the 39th Annual Hawaii International
Conference on, 2006. IEEE, 97c-97c.

11. DUARTE, A. I. M. & COSTA, C. J. 2012. Information systems: life cycle and success. Proceedings of the
Workshop on Information Systems and Design of Communication. Lisbon, Portugal: ACM.

12. GLASGOW, R. E. 2007. eHealth evaluation and dissemination research. Am J Prev Med, 32, S119-26.
13. HALLIKAINEN, P. & CHEN, L. 2006. A Holistic Framework on Information Systems Evaluation with a
Case Analysis. Electronic Journal of Information Systems Evaluation, 9.

14. HICKS, L. L. & BOLES, K. E. A comprehensive model for evaluating telemedicine. Studies in Health
Technology and Informatics, 2004. 3-13.

15. JEN-HWA HU, P. Evaluating telemedicine systems success: a revised model. System Sciences, 2003.
Proceedings of the 36th Annual Hawaii International Conference on, 2003. IEEE, 8 pp.

16. KAMSU-FOGUEM, B. 2014. Systemic modeling in telemedicine. European Research in Telemedicine /

La Recherche Europenne en Tlmdecine, 3, 57-65.

17. KIDHOLM, K., EKELAND, A. G., JENSEN, L. K., RASMUSSEN, J., PEDERSEN, C. D., BOWES, A.,
FLOTTORP, S. A. & BECH, M. 2012. A model for assessment of telemedicine applications: mast.
International journal of technology assessment in health care, 28, 44-51.

18. LE GLDIC, S., FOURNIER, M., MALEK, S. & DUPRAZ-POISEAU, A. Telemedicine and e-health
systems: Regulatory framework and issues. 2011 the 14th International Symposium on Wireless Personal
Multimedia Communications: Communications, Networking and Applications for the Internet of Things,
WPMC'11, 2011.

19. MATHUR, A., KVEDAR, J. C. & WATSON, A. J. 2007. Connected health: A new framework for
evaluation of communication technology use in care improvement strategies for type 2 diabetes.
Current Diabetes Reviews, 3, 229-234.

20. MOHD AL-ADAILEH, R. 2009. An evaluation of information systems success: A user perspective-The
case of Jordan telecom group. European Journal of Scientific Research, 37, 226-239.

21. MOHAMMED, S. A. & YUSOF, M. M. 2013. Towards an evaluation framework for information quality
management (IQM) practices for health information systems evaluation criteria for effective IQM
practices. Journal of Evaluation in Clinical Practice, 19, 379-387.

22. MOHD, M., PAUL, R. J. & STERGIOULAS, L. K. Towards a Framework for Health Information Systems
Evaluation. HICSS, 2006.

23. O'NEILL, S. A., NUGENT, C. D., DONNELLY, M. P., MCCULLAGH, P. & MCLAUGHLIN, J. 2012.
Evaluation of connected health technology. Technology and Health Care, 20, 151-167.

24. OHINMAA, A. 1997. A model for the assessment of telemedicine and a plan for testing of the model
within five specialities, Stakes.

25. PARYANI, S. A framework for evaluation of telemedicine. HEALTHCOM 2006: Mobile E-Health for
Developing Countries - 2006 8th International Conference on e-Health Networking, Applications and
Services, 2006. 202-205.

26. PAVEL, M., JIMISON, H. B., WACTLAR, H. D., HAYES, T. L., BARKIS, W., SKAPIK, J. & KAYE, J.
2013. The Role of Technology and Engineering Models in Transforming Healthcare. Biomedical
Engineering, IEEE Reviews in, 6, 156-177.

27. RILEY, W. T. Evaluation of mHealth. Institute of Medicine and National Research Council,
Communications and technology for violence prevention: Workshop summary, 2012. National Academies,
72-86.

28. ROINE, R., OHINMAA, A. & HAILEY, D. 2001. Assessing telemedicine: a systematic review of the
literature. Canadian Medical Association Journal, 165, 765-771.

29. SARKAR, B. B., SANYAL, S. & CHAKI, N. 2011. Distributed framework for tele health monitoring
system. Proceedings of the 1st International Conference on Wireless Technologies for Humanitarian
Relief. Amritapuri, Kollam, Kerala, India: ACM.

4
6

30. SRUR, B. L. & DREW, S. Challenges in designing a successful e-health system for Australia. Information
Technology in Medicine and Education (ITME), 2012 International Symposium on, 3-5 Aug. 2012 2012.
480-484.

31. STRYKHALIUK, B., KOLODIJ, R. & OLEXIN, M. Set-theoretic model of the telemedicine with patient
state consideration. Modern Problems of Radio Engineering, Telecommunications and Computer Science Proceedings of the 11th International Conference, TCSET'2012, 2012. 344-345.

32. THIYAGARAJAN, C. & CLARKE, M. A systematic review of technical evaluation in telemedicine

systems. Engineering in Medicine and Biology Society, 2006. EMBS'06. 28th Annual International
Conference of the IEEE, 2006. IEEE, 6320-6323.

33. TOPACAN, U., BASOGLU, A. N. & DAIM, T. U. Exploring the success factors of health information
service adoption. Management of Engineering & Technology, 2008. PICMET 2008. Portland International
Conference on, 27-31 July 2008 2008. 2453-2461.

34. VIMARLUND, V. & LE ROUGE, C. Barriers and opportunities to the widespread adoption of
telemedicine: A Bi-country evaluation. Studies in Health Technology and Informatics, 2013. 933.

35. WOOD, S., GANGADHARAN, S., TYRER, F., GUMBER, R., DEVAPRIAM, J., HIREMATH, A. &
BHAUMIK, S. 2014. Successes and Challenges in the Implementation of Care Pathways in an
Intellectual Disability Service: Health Professionals' Experiences. Journal of Policy and Practice in
Intellectual Disabilities, 11, 1-7.

36. WRIGHT, A. & SITTIG, D. F. 2008. A framework and model for evaluating clinical decision support
architectures. Journal of Biomedical Informatics, 41, 982-990.

37. YA-LI, Z., XIAO-RONG, D., POON, C. C. Y., LO, B. P. L., HEYE, Z., XIAO-LIN, Z., GUANGZHONG, Y., NI, Z. & YUAN-TING, Z. 2014. Unobtrusive Sensing and Wearable Devices for Health
Informatics. Biomedical Engineering, IEEE Transactions on, 61, 1538-1554.

38. YUCEL, G., CEBI, S., HOEGE, B. & OZOK, A. F. 2012. A fuzzy risk assessment model for hospital
information system implementation. Expert Systems with Applications, 39, 1211-1218.

39. YUSOF, M. M., KULJIS, J., PAPAZAFEIROPOULOU, A. & STERGIOULAS, L. K. 2008a. An

evaluation framework for Health Information Systems: human, organization and technology-fit factors
(HOT-fit). International Journal of Medical Informatics, 77, 386-398.

40. YUSOF, M. M., PAPAZAFEIROPOULOU, A., PAUL, R. J. & STERGIOULAS, L. K. 2008b.

Investigating evaluation frameworks for health information systems. International Journal of
Medical Informatics, 77, 377-385.

4
7

Appendix B
Code for Bubble plot (Figure 12)
%AUTHOR: PRIYANKA.V
%Date:8/7/2014 %UNIVERSITY
OF LIMERICK

%TITLE: PLOTTING OF CUSTOM GRAPHS WITH MARKERS

%Runs with matlabr2011 and higher versions.
%statement of responsibilty: Priyanka.V
%REFERENCES:[1] www.mathworks.com
% [2]
https://admin.kuleuven.be/icts/onderzoek/wetsoft/software/matlab/downloads/matlabcoursesessi
on3slides
%****************************************************************************
%Start of code
%****************************************************************************
clc;
clear all;
close all;

%defining the position of the

axis %defining X-axis
x = [0 1 1 1 1 1 2 2 3 3 3 3 4 4 4 4 4 5 5
5 ]; %defining Y-axis
y = [0 1 2 3 4 5 2 4 2 3 4 5 1 2 3 4 7 2 5 9 ];
%Defining the size of the marker according to the position of X axis where
%the s depends on value power2.
s= [36 150 350 250 250 150 150 150 200 200 150 150 200 300 350 250 150 250 150 150];
%defining individual positions to create new overlapping on the existing
%scatter plot according to the required parameters
x0 = [0]; y0 = [0];

48

x1 = [1]; y1= [1]; y2 = [2]; y3 = [3]; y4 = [4]; y5 =

[5]; x2 = [2]; y2 = [2]; y4 = [4];
x3 = [3]; y2 = [2]; y3 = [3]; y4 = [4]; y5= [5];
x4 = [4]; y1 = [1]; y2 = [2]; y3 = [3]; y4 = [4];y7 =
[7]; x5 = [5]; y2 = [2]; y5 = [5]; y9 = [9];
x6 = [6]; y6 =
[0]; c = [0 0 0];
%plots the bubble plot or scatter plot according the X and Y axis and S
%specifies the size.
%see [1] scatter(x,y,s,c);
xlabel('Research facet');
ylabel('context facet');

title('Validation of a Systematic Map in the form of a Bubble Plot');

hold on
plot(x6,y6);
%writes the required quantity of papers on custom positions
%see [2]
text(x1,y1,'1');
text(x1,y2,'5');
text(x1,y3,'3');
text(x1,y4,'3');
text(x1,y5,'1');
text(x2,y2,'1');
text(x2,y4,'1');
text(x3,y2,'2');
text(x3,y3,'2');
text(x3,y4,'1');
text(x3,y5,'1');
text(x4,y1,'2');
text(x4,y2,'4');

4
9

text(x4,y3,'5');
text(x4,y4,'3');
text(x4,y7,'1');
text(x5,y2,'3');
text(x5,y5,'1');
text(x5,y9,'1');
%see[2]
grid on
%****************************************************************************
%End of code
%****************************************************************************
Code for Bubble Plot (Figure 13)
%AUTHOR: PRIYANKA.V
%Date:8/7/2014 %UNIVERSITY
OF LIMERICK

%TITLE: PLOTTING OF CUSTOM GRAPHS WITH MARKERS

%Runs with matlabr2011 and higher versions.
%statement of responsibilty: Priyanka.V
%REFERENCES:[1] www.mathworks.com
% [2]
https://admin.kuleuven.be/icts/onderzoek/wetsoft/software/matlab/downloads/matlabcoursesessi
on3slides
%****************************************************************************
***************
%start of code
%****************************************************************************
***************
clear all;
close all;
%defining the position of the axis

5
0

%defining X-axis
x = [0 1 1 1 1 1 1 1 2 2
2 ]; %defining Y-axis
y = [0 1 2 3 4 5 7 9 2 3 4 ];
%Defining the size of the marker according to the position of X axis where
%the s depends on value power2.
s= [36 150 700 350 400 200 100 100 100 250 100 ];

%defining individual positions to create new overlapping on the existing

%scatter plot according to the required parameters
x0 = [0]; y0 = [0];
x1 = [1]; y1 = [1]; y2 = [2]; y3 = [3]; y4 = [4]; y5 = [5]; y7 = [7]; y9 = [9];
x2 = [2]; y2 = [2]; y3 = [3]; y4 = [4];
x6 = [3]; y0 =
[0]; c = [0 0 0];
%plots the bubble plot or scatter plot according the X and Y axis and S
%specifies the size.
%see [1] scatter(x,y,s,c);
xlabel('Research facet');
ylabel('context facet');

title('Validation of the Systematic Map in the form of a Bubble plot');

hold on
plot(x6,y0);
%writes the required quantity of papers on custom positions
%see [2]
text(x1,y1,'2');
text(x1,y2,'14');
text(x1,y3,'6');
text(x1,y4,'8');
text(x1,y5,'3');

5
1

text(x1,y7,'1');
text(x1,y9,'1');
text(x2,y2,'1');
text(x2,y3,'4');
text(x2,y4,'1');
%see[2]
grid on
%****************************************************************************
%End of code
%****************************************************************************

5
2