User Experience and User Acceptance

of an Augmented Reality Based

Knowledge-Sharing Solution in Industrial
Maintenance Work

Susanna Aromaa(&), Antti Väätänen, Mika Hakkarainen,

and Eija Kaasinen

VTT Technical Research Centre of Finland Ltd,

P.O. Box 1300, 33101 Tampere, Finland
{Susanna.Aromaa,Antti.Vaatanen,Mika.Hakkarainen,
Eija.Kaasinen}@vtt.fi

Abstract. In industrial service business, maintenance processes can be chal-

lenging due to their complexity and knowledge intensiveness. The goal of this
study was to evaluate the user experience and user acceptance of an augmented
reality (AR) based, knowledge-sharing system, in industrial maintenance. An
augmented reality system was evaluated in a focus group and a field study. The
results indicate that the user experience was positive and the AR system was
well accepted. The participants felt that the AR system was useful for their work
and it supported information and knowledge sharing. The social media features
for adding and sharing one’s own comments, notes and pictures were seen as
very useful. However, some concerns regarding the quality and accuracy of the
information content were raised. This study provides insights into AR-based
knowledge sharing in industrial maintenance; however, the results can also be
utilized in other areas, such as manufacturing.

Keywords: Knowledge sharing
Augmented reality
User experience

Industrial maintenance

1 Introduction

Industrial machine manufacturers are starting to invest more in their service business,
as global customers expect high quality lifecycle services. However, maintenance
processes can be challenging, as the work machines are complex and often customized,
their lifecycle is long and there is a lot of information connected to the machines, which
includes both initial product data, and maintenance and usage history. In addition, there
can be knowledge related to the machines and their maintenance that is tacit knowl-
edge, possessed by experienced maintenance personnel. Challenges can originate, for
example, from inadequacy of formal documentation and non-available up-to-date
information. Augmented reality (AR) and advanced communication technologies are
promising solutions for supporting knowledge sharing in industrial maintenance work,

© Springer International Publishing AG 2018

T. Ahram and C. Falcão (eds.), Advances in Usability and User Experience,
Advances in Intelligent Systems and Computing 607, DOI 10.1007/978-3-319-60492-3_14
146 S. Aromaa et al.

as they can provide fluent access to contextually relevant knowledge, in different

maintenance situations.
The purpose of this study was to evaluate a user experience (UX) and a user
acceptance of an AR technology-based knowledge-sharing solution, in maintenance
work. The paper is organized as follows. Section 2 describes the background. Section 3
presents the AR system and Sect. 4 describes the user studies. The results are presented
in Sect. 5, discussed in Sects. 6 and 7 concludes the research.

2 Background

The European standard [1] defines maintenance as a ‘combination of all technical,

administrative and managerial actions during the lifecycle of an item intended to retain
it in, or restore it to, a state in which it can perform the required function’. Maintenance
work can include different type of activities, such as, scheduled operations, unsched-
uled operations, inspections and testing [2].
Industrial maintenance is complex and knowledge intensive work. The work
includes both tacit and explicit knowledge. Explicit knowledge is more visible and is
transmittable in formal language [3]. Tacit knowledge is more “hidden” and can be
rooted in action, in a specific context [3]. Augmented reality means that virtual objects
are superimposed, on top of real objects, in a real environment [4]. Augmented reality
can, thus, make information contextual by linking it to the objects of the physical world
and making AR a promising solution to support both explicit and tacit knowledge
sharing. The use of AR technologies in maintenance has been examined in many
studies [5–7]. In addition, similar studies have been conducted in design and manu-
facturing [8–12].
In our earlier research, we studied the use of AR technologies and wearables, in real
maintenance contexts [13–15]. This paper extends those studies by proposing social
media features to support peer-peer knowledge sharing. Social media can be defined as
a combination of various forms of media content that are publicly available in a
platform and that are created by end-users in a participatory and collaborative fashion
[16]. Kaplan and Haenlein [16] propose a two-dimensional classification for social
media: presence/media richness and self-disclosure/self-presentation. In [17] an AR 2.0
concept is presented that allows active user participation in the content creation pro-
cess. With the mobile AR system, non-expert users could create and share AR content,
at specific locations. Our work is related to their work, but we have targeted closed
professional communities, while [17] targeting an open social AR for consumer use.

3 System Description

An Android smart phone was used as the platform device. The AR system utilized both
marker based and planar image based tracking approaches. Tracking was done with the
VTT’s ALVAR SDK for Augmented Reality [18]. The AR demo illustrated the status
information of an electrical system (e.g., condition, fault codes) and enabled social
media features for sharing notes and pictures with other service personnel.
 User Experience and User Acceptance of an Augmented Reality 147

The marker on top of the cover was used to visualize the runtime information from
the system and to show a virtual view inside the electrical box (Fig. 1, left-hand side).
The user was able to switch between these two views. The interior of the electrical box
was tracked by using the planar image tracking. The system recognized the content and
visualized user clickable hotspots, on top of the predefined components. The user was
able to select any of the hotspots and the system provided more information on the
selected component and/or system (Fig. 1, right-hand side).

Fig. 1. Two screenshots of the AR system [19]. The runtime information on the cover of the
electrical box (on the left-hand side). Detailed information of the component on top of the open
electrical box (on the right-hand side).

The social media service was implemented on a proprietary local server running on
a PC. Android smart phones were connected to the server by using the WiFi hotspot of
the PC. The server collected the notes, comments and remarks of the user and shared
this information with the other participants. The user was also able to retrieve and store
additional runtime information, for the system.
The user was able to take and share a photo of the target that he/she wanted to write
a social media note about. When composing the note, the user was able to highlight the
point on the image, define the category and write textual information with keywords.
Hashtags were used for the identification of keywords. Users were able to search
messages using these keywords.

4 User Studies

User studies were made at a global industrial manufacturer and maintenance company,
which provides global industrial maintenance services for cranes and machine tools.
The company was actively developing their AR system with the research partner and
provided the possibility to interview crane professionals. Two different setups were
used for the AR system evaluation: a focus group and a field study. The same AR
system was used in both studies. This chapter describes the participants, data collection
methods and test procedures.
148 S. Aromaa et al.

4.1 Data Collection

In both studies, quantitative data was collected with a questionnaire and qualitative data
by interviewing and observing. The questionnaire was divided into two sections. In the
first section, user experience was evaluated with smiley faces and the Questionnaire for
User Interaction Satisfaction (QUIS) [20]. In the second part, there were 14 questions
related to knowledge sharing and user acceptance. The user acceptance questions were
adopted from the System Usability Scale (SUS) [21] and the Technology Acceptance
Model (TAM) [22]. Smiley faces and user acceptance questions were answered by
using a 5-point Likert scale, ranging from strongly agree (5) to strongly disagree (1).
The QUIS-based questions used a 9-point Likert scale, ranging from really good (9) to
really bad (1). Interview themes were related to development ideas for the system, how
suitable it was for doing maintenance work and how it would change the work.
Photographs and videos were taken from the user studies and the conversations were
recorded via voice recorder. A transcription was made from the recordings, by a
subcontractor. A thematic analysis [23] was used for data analyzation.

4.2 Focus Group

Eight people from the crane company participated in the focus group (Fig. 2). There
were two maintenance technicians, a trainer, a team leader and four managers (product,
sales, maintenance and department). The average age was 38 years (age range: 30–44
years) and the average time of employment, within the current company, was 9 years
(range: 1–23 years). All participants were active users of smart phones. Six of them had
some experience with AR, three of them knew of the term, AR, and one did not have
any experience with AR. Five similar Android smartphones, with the AR application,
were provided for the evaluation. In addition, a real crane electrical box and printed
pictures of it were used to superimpose augmented information (Fig. 2).

Fig. 2. Testing the AR system with the focus group.

 User Experience and User Acceptance of an Augmented Reality 149

First, an introduction on using the AR system was provided, consent forms were
signed and participant demographics were collected with a questionnaire. Afterwards,
participants were encouraged to try the AR system on their own. The users were guided
through the process of the following tasks:
– Creating a username and login to the AR system,
– Using AR features for viewing runtime information from the electrical box cover,
– Using the eye icon, in the lower right corner, to virtually view the inside of the
electrical box,
– Trying social media features and browsing comments and conversations by clicking
the speech balloon on the right side,
– Opening the electrical box and monitoring AR visualizations of the components
(Fig. 3),
– Selecting an AR marked component and writing a note, including a keyword with a
hashtag sign, e.g., #contactor is broken (Fig. 3),
– Searching for the keyword on the social media notes section,
– Freely using the systems.

Fig. 3. The augmented reality visualization on top of the electrical box and writing a comment
using the social media tool.

After using the AR system, participants filled in the questionnaire. Then, the
interview questions were asked and discussed, as a group. The whole process took
approximately two hours.

4.3 Field Study

Two crane maintenance technicians (aged 29 and 32 years, with 10 and 8 years of
maintenance work experience, respectively) took part in the field study. Both were
active users of smart phones and they had some experience with AR solutions. First, an
introduction to the AR system use was provided, consent forms were signed and
participant demographics were collected with the questionnaire. Both technicians were
given a test smartphone and the test tasks and phases were same as in the focus group
session. Then, both maintenance technicians climbed into a maintenance target crane,
with the test electrical box. In the crane, the participants could tryout the system, in a
150 S. Aromaa et al.

real context (Fig. 4). During the test session, the participants were encouraged to ask
questions and they were provided with support in using the AR system. After testing
the AR system, the technicians were interviewed in their office. Both maintenance
technicians filled in the questionnaire, first, and then they were interviewed together.

Fig. 4. Field study with the AR system on top of the real crane.

5 Results

The results from the focus group and the field study are presented together. Therefore,
the number of participants is 10, in the following subchapters.

5.1 User Experience and User Acceptance

The AR system user experience was rated as good, with smiley faces (M = 4.2,
STDEV = 0.42). The overall user interaction satisfaction was also good (Fig. 5).
Participants agreed that interaction with the system was easy and stimulating. The
participants also liked the symbols and overall visual appearance of the AR system. In
addition, the system was rated as flexible, effective, wonderful and satisfying to use.

Fig. 5. The overall user interaction satisfaction of the AR system.

 User Experience and User Acceptance of an Augmented Reality 151

Ease of use was evaluated as being good, in the AR system (Fig. 6). The partici-
pants strongly agreed that the system was easy to learn and use: “It was simple to use”.
They also agreed that information was easy to find and that they would like to use the
AR system, in the future. The participants did not feel physical fatigue (e.g., hands,
neck or eyes) after using the AR system. In addition, they stated that the system was
quite reliable.

Fig. 6. Ease of use of the AR system.

The perceived usefulness was also evaluated as being good (Fig. 7). However, it
could be seen that there were more doubts as to whether the system would really be
applicable in a real maintenance context: “The use of the AR system should decrease
the work and not double it”. They agreed that the AR system would make it easier to do

Fig. 7. Usefulness of the AR system.

152 S. Aromaa et al.

a job. They also agreed that the AR system would enable them to accomplish tasks
more quickly, and increase the quality of their work.
Results from the questions related to the information and knowledge sharing fea-
tures of the AR system are presented in Fig. 8. Attitudes toward knowledge and
information sharing were positive. It seems, though, that the adequacy of the infor-
mation could have been better. The participants were willing to share information with
others and thought that sharing notes and comments with others is useful (M = 5.0,
STDEV = 0.00). One participant mentioned that helping and guiding colleagues, when
needed, is typically rewarding and that the attitude toward helping and sharing
knowledge is very positive. The tagged keywords were also found to be useful. The
participants also indicated that they would like to supplement the text notes with
photographs, videos and audio.

Fig. 8. Usefulness of knowledge sharing with the AR system.

5.2 Possibilities
The participants thought that by using the AR system, the information (e.g., manuals)
would always be available (e.g., no need to climb down from the crane and go to a car
to read a manual). It could make work more automatic and maintenance technicians
could “concentrate on the real work”. The AR system could guide users through work
processes and ensure better work quality.
Participants felt that the use of the AR system could improve the quality of the
maintenance reports. Currently, there are variations in report quality between main-
tenance technicians. The participants thought that, with the AR system, they could
supplement and enhance written texts with photographs and videos.
 User Experience and User Acceptance of an Augmented Reality 153

The participants said that the AR system makes it easier to record information that
maintenance technicians know to be important. Therefore, the AR system could pro-
vide more important and situationally relevant information to the maintenance tech-
nicians (e.g., maintenance technicians would be able to see who had done previous
maintenance work on a specific machine). This could also be a solution for gathering
and sharing more knowledge from other companies’ machines that the technicians are
also maintaining.
If the AR system uses location tracking, it could automatically provide needed
information to the maintenance technician (e.g., information about the crane). A lo-
cating feature would also help to provide information on specific machines. The par-
ticipants emphasized that it is useful to know what kinds of maintenance actions have
been done and if the machine requires any specific extra inspections or operations. It
could also make the preparation phase faster and easier. Customer service could be
improved by providing on-time information and faster reporting of maintenance tasks.

5.3 Challenges
The participants felt that the system needs to be easy to use for maintenance techni-
cians. They thought that user acceptance might be hindered with some maintenance
technicians, due to their attitudes towards new technologies, lack of knowledge on how
to use the system, lack of perceiving any added value and language barriers. However,
the participants thought that some technicians would be keen to use the system
immediately. One participant estimated that one third of the technicians would be very
willing to use the new system, one third would consider utilizing this kind of new AR
technologies and one third would resist it.
The participants were also concerned that the AR system would become just
another system they needed to use, adding tasks and responsibilities to their current
work. Participants had doubts about the robustness and operability of the AR system in
harsh environments (e.g., high moisture, dust, noise, heat, cold). In some specific
maintenance locations, it is not possible to take smart phones with you, at all.
Regarding the social media features, the participants discussed who would really be
responsible for the quality of the information and how trustworthy the information
would be if anyone can create and share it. When maintenance technicians create the
information, it is possible that it is false, contradictory or there is too much of it. In
addition, this kind of approach allows the sharing of false work procedures (e.g., how to
bypass some errors with dangerous/safety critical ways). The participants said that
some kind of moderation might be good to have in place for analyzing the content
before publishing it.
The participants discussed that their products and systems are so complex that the
AR system might be too simple to manage all of the information. They also wondered
at what level the information should be attached to something (“Will the added text be
attached to a component, a part, a machine or a factory”). The accuracy of the infor-
mation was also discussed (e.g., if the AR system can give location information, how
exact is the location, really). It is also important to be able to integrate the AR system
with current maintenance information systems.
154 S. Aromaa et al.

Other concerns included, that it can happen that there is no online connection
available. And, in some countries, sending videos might be expensive. The participants
also highlighted uncertainty about the technology: What if does not work? How would
they be able to operate, then?

6 Discussion

In general, the user experience and user acceptance of the AR system was positive, and
they thought that the system could support knowledge sharing in maintenance work.
Issues that could be obstacles for the deployment of the AR system were related to the
user, the technology, the environment/task and information. User challenges were
related to the acceptability, usability and usefulness of the product (e.g., does it give
added value to users). The technology challenges concerned integration with current
systems, data transfer and on-line connection possibilities. The environment and task
challenges mainly related to the robustness and reliability of the AR system in harsh
environments. The information challenges related to issues, such as, what information
is needed, how it is presented, is it contradictory, how much information is there and
how to make sure that information is adequate and reliable.
By using the AR system, it is possible to support the “externalization” mode of
Nonaka’s [3] theory, in particular, in which tacit knowledge is transferred to explicit.
For example, a maintenance technician knows information that is related to only one
machine or one type of the machine. The maintenance technician can use a hashtag in
the AR system to add, for example, knowledge of a different oil type that it needs. This
kind of information is not necessarily shared, otherwise.
In the classification of social media by Kaplan and Haenlein [16], the AR system
could be set on collaborative projects that are low in self-presentation/self-disclosure
and low in social presence and media richness. One interesting future research question
related to this is: Do the workers want to extend their social media dimensions in the
work context (e.g., does a maintenance technician want to be more open with their
personal information with others and/or do they want to represent themselves as avatars
in a virtual world).
Issues that could have had an effect on results were that the AR system was a
prototype that the participants only tried out for short periods of time and the studies
were only carried out within a single company. In addition, the AR system was not
integrated into the information systems in use and it was not used to complete real work
tasks. Longer use of the AR system in a real usage context might have supplement the
results. Our next research steps will be to implement the AR system in real prolonged
use, with a more versatile set of data and tasks. Then, it would be possible to evaluate it
under real use for information sharing as well as its effects on ergonomics and safety
issues.
 User Experience and User Acceptance of an Augmented Reality 155

7 Conclusions

This paper represents results from a user study of an augmented reality (AR) based
knowledge-sharing solution in industrial maintenance. The AR system was evaluated
in a focus group session, with different company stakeholders and by two maintenance
technicians in a real maintenance environment.
The user experience and user acceptance of the demonstrated AR system was
evaluated as being good and exploitable. Technically, the AR recognition worked well
and it was able to provide applicable information for maintenance tasks. The partici-
pants liked the knowledge sharing and social media features of the AR system.
However, participants still had concerns related to the quality and reliability of the
information that maintenance technicians can create and share, themselves. They also
had doubts as to whether the AR system would really bring added value, in practice.
This study was made in the context of industrial maintenance; however, the results are
also applicable to other AR technology usage areas, such as design and manufacturing.

Acknowledgments. This study was funded by Tekes – the Finnish Funding Agency for
Innovation – and was carried out as part of the S-STEP research project, under DIMECC (Digital,
Internet, Materials & Engineering Co-Creation ecosystem). The authors are grateful to all of the
researchers and company representatives who contributed to and supported the work presented in
this publication.

References
1. European Committee for Standardization: EN 13306. Brussels (2010)
2. Reason, J.: Managing the Risks of Organizational Accidents. Ashgate Publishing Ltd.,
London (1997)
3. Nonaka, J.: A dynamic theory of organizational knowledge creation. Organ. Sci. 1(5), 14–37
(1994)
4. Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., MacIntyre, B.: Recent advances
in augmented reality. Comput. Graph. Appl. 21(6), 34–47 (2001)
5. Kunze, K., Wagner, F., Kartal, E.: Does context matter? -A quantitative evaluation in a real
world maintenance scenario. In: Pervasive Computing, pp. 372–389 (2009)
6. Henderson, S., Feiner, S.: Exploring the benefits of augmented reality documentation for
maintenance and repair. IEEE Trans. Vis. Comput. Graph. 17(10), 1355–1368 (2011)
7. Henderson, S.J., Feiner, S.: Evaluating the benefits of augmented reality for task localization
in maintenance of an armored personnel carrier turret. In: Science and Technology
Proceedings - IEEE 2009 International Symposium on Mixed and Augmented Reality,
ISMAR 2009, pp. 135–144 (2009)
8. Nee, A.Y.C., Ong, S.K., Chryssolouris, G., Mourtzis, D.: Augmented reality applications in
design and manufacturing. CIRP Ann. - Manuf. Technol. 61(2), 657–679 (2012)
9. Ong, S.K., Yuan, M.L., Nee, A.Y.C.: Augmented reality applications in manufacturing: a
survey. Int. J. Prod. Res. 46(10), 2707–2742 (2008)
10. Tang, A., Owen, C., Biocca, F., Mou, W.: Comparative effectiveness of augmented reality in
object assembly. In: Proceedings of the Conference on Human Factors in Computing
Systems CHI 2003, pp. 73–80. ACM Press (2003)
156 S. Aromaa et al.

11. Schwerdtfeger, B., Klinker, G.: Supporting order picking with augmented reality. In: 7th
IEEE International Symposium on Mixed and Augmented Reality 2008, ISMAR 2008,
pp. 91–94 (2008)
12. Wang, X., Ong, S.K., Nee, A.Y.C.: A comprehensive survey of augmented reality assembly
research. Adv. Manuf. 4(1), 1–22 (2016)
13. Aromaa, S., Aaltonen, I., Väätänen, A.: Technology concepts to improve knowledge sharing
during maintenance. In: The Ninth International Conference on Advances in
Computer-Human Interactions (ACHI 2016) (2016)
14. Aromaa, S., Väätänen, A., Aaltonen, I., Heimonen, T.: A model for gathering and sharing
knowledge in maintenance work. In: Proceedings of the 33rd Annual Conference of the
European Association of Cognitive Ergonomics (ECCE 2015) (2015)
15. Aromaa, S., Aaltonen, I., Kaasinen, E., Elo, J., Parkkinen, I.: Use of wearable and
augmented reality technologies in industrial maintenance work. In: Proceedings of the 20th
International Academic Mindtrek Conference, pp. 235–242. ACM (2016)
16. Kaplan, A.M., Haenlein, M.: Users of the world, unite! the challenges and opportunities of
social media. Bus. Horiz. 53(1), 59–68 (2010)
17. Langlotz, T., Mooslechner, S., Zollmann, S., Degendorfer, C., Reitmayr, G., Schmalstieg,
D.: Sketching up the world: In-situ authoring for mobile augmented reality. Pers. Ubiquit.
Comput. 16(6), 623–630 (2012)
18. Alvar Tracking SDK. http://virtual.vtt.fi/virtual/proj2/multimedia/alvar/index.html
19. Aromaa, S., Honkamaa, P., Kaasinen, E., Väätänen, A., Valtonen, J., Rauhala, V., Niemelä,
A., Blomberg, P.: Knowledge sharing solutions for field service personnel - contextual
guidance. In: DIMECC Oy (2017)
20. Chin, J.P., Diehl, V.A., Norman, K.L.: Development of an instrument measuring user
satisfaction of the human-computer interface. In: Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems, pp. 213–218. ACM (1988)
21. Brooke J.: SUS-A quick and dirty usability scale (1996)
22. Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information
technology. MIS Q. 13(3), 319–340 (1989)
23. Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol. 3(2), 77–
101 (2006)