Developing a Questionnaire for Measuring Mobile Business Service Experience

Maiju Vuolle
Department of Business Information Management and Logistics Tampere University of Technology P.O.Box 541 33101 Tampere, Finland +358 40 559 6424

Titti Kallio
TeliaSonera P.O.Box 585 00051 Sonera, Finland +358 40 503 6957

Minna Kulju
VTT, Media and Mobile usability Technical Research Centre of Finland P.O. Box 1300 33101 Tampere, Finland +358 20 722 3386

titti.kallio@teliasonera.com

maiju.vuolle@tut.fi Mari Tiainen

Nokia Oyj, Services & Software Visiokatu 4 33720 Tampere, Finland +358 50 384 4106

minna.kulju@vtt.fi Teija Vainio

Department of Software Systems Tampere University of Technology P.O. Box 589 33101 Tampere, Finland +358 40 849 0729

Heli Wigelius
Department of Software Systems Tampere University of Technology P.O. Box 589 33101 Tampere, Finland +358 40 500 6104

mari.tiainen@nokia.com ABSTRACT

teija.vainio@tut.fi
In this paper, three dimensions are conceptualized to represent elements of mobile business service experience. By combining these perspectives, namely usability, mobile working context and mobile work productivity, we aim to understand the nature of mobile work and how mobile business services could support users in this context. A questionnaire, MoBiS-Q, for measuring these dimensions has been developed and tested in three pilot studies during real service development processes. Iterative item generation and refinement were conducted through examination of the literature, interviews and pre-testing. MoBiS-Q is a multidisciplinary tool that provides a basis for joint development between relevant parties and departments in an organization, including user representatives, sales, marketing, product management, technology, and usability.

heli.wigelius@tut.fi

1. INTRODUCTION
Mobile business services are increasingly being used for work purposes in several domains, such as the police [17], independent consultants [20], field force [35], sales and marketing [44]. These different activities have various requirements for both mobility and services supporting mobile work tasks [5]. In this paper, the concept of a mobile business service refers to a business-toemployee service that the user operates using a portable device. Examples of such services include business data services that provide access to corporate intranet and email, and services that enable data collection in the field (see, e.g., [35]). In order to develop successful mobile business services, there is a need to understand the dimensions of success. In earlier studies, perceived ease of use and perceived usefulness have been emphasized as the most critical factors for technology acceptance in the work context [6]. In addition, the usability of a mobile business service has been seen as an important driver of the business impacts of using the service [31]. However, examining the customer experience of mobile business services is challenging because user experience consists of many elements, not only technology. In addition, in the area of business services, to be a critical factor, user experience needs to be combined with enhancements in work productivity. A major challenge for improvements here is also that customer experience is dependent on a network of technology and service providers. There is no solid way to evaluate if a mobile business service is appropriate both from a perspective of usability and productivity impacts. Conducting traditional usability and productivity evaluations for mobile business services increases the risk of irrelevant results by failing to consider the context of mobile work. Thus, this paper focuses on the evaluation of experience including usability and productivity factors for mobile business services. The purpose of this paper is to investigate the dimensions that determine users mobile business service

Categories and Subject Descriptors

H.5 [Information interfaces and presentation (e.g., HCI)]: User interfaces theory and methods, Evaluation/methodology; H.4 [Information systems applications]: miscellaneous

General Terms
Measurement, Human Factors, Performance, Economics, Management

Keywords
Evaluation, Questionnaire, Usability, Mobile work context, Productivity, Mobile business service

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. MobileHCI 2008, September 25, 2008, Amsterdam, the Netherlands. Copyright 2008 ACM 978-1-59593-952-4/08/09$5.00.

FP

53

experience and to develop a questionnaire for measuring that experience. This questionnaire is meant for service developers for benchmarking and evaluating the success of the service against competitors, for evaluating the new versions of the service, and for evaluating the benefits the service could provide for end users and companies deploying the service. While the focus of this paper is to develop and evaluate a specific tool, our findings will also be of interest to those concerned with evaluation of usability and productivity within mobile business services in general. The primary contribution of this paper is to provide a novel multidisciplinary tool for evaluating key usability and productivity issues based on individual workers subjective views on these issues. The evaluation tool is called mobile business service questionnaire (MoBiS-Q). To develop the questionnaire, we iteratively generated and refined items based on the literature, existing case studies ([30]; [31]), and interviews. In order to evaluate MoBiS-Q, we conducted three pilot tests in three different companies with service developers and real end-users. A secondary contribution of this paper is to emphasize what kind of challenges there are in integrating the evaluation results in the actual development process of a mobile business service. The paper proceeds as follows: the following section provides some background for measuring usability and impacts in general and of earlier studies of mobile commerce and systems. Section 3 focuses on the evaluation of mobile business service experience and describes the concept and dimensions of mobile business service experience in more detail. Section 4 presents the iterative development of our evaluation tool MoBiS-Q, and Section 5 outlines our findings. The final section concludes.

of mobile applications. These challenges include, for example, mobile context, multimodality, connectivity, small screen size, different display resolutions, limited processing capability and power, restrictive data entry methods [51], and the way the mobile context is connected to network, or other, more traditional working environments. In addition, earlier research has focused mainly on mobile consumer services and not on employee services. Thus, the characteristics of mobile work are not emphasized in existing questionnaires and models. For example, in order to be more suitable for consumer services, TAM has been extended with innovation diffusion theory, perceived risk and cost [48] as well as value, trust and ease of adoption [19]. Moreover, in order to explain customers mobile Internet adoption, Kim, Chan and Gupta have developed the Value-based Adoption Model, which is also based on TAM [25]. In this model, the perceived value of mobile Internet is determined by usefulness, enjoyment, fee and technicality. Usage fee and enjoyment are important issues in consumer settings because consumers adopt and use mobile services voluntarily for personal purposes. The work context is somewhat different as employees are using mobile business services for work purposes and the cost of mandatory adoption and usage is born by the organization. [25] As companies pay for mobile business services, they also want some return on their investments. Thus, the adoption and use of mobile business services must also have individual and organizational impacts. The linkage between technology and individual performance is addressed, for example, in the model of information system (IS) success [7], [8]. According to the IS success model, system quality, information quality and service quality affect intention to use, or use and user satisfaction. In addition, the amount of use may have a positive or negative effect on the degree of user satisfaction and vice versa. Moreover, the use and user satisfaction lead to an individual impact, and this impact should eventually have some organizational influence. Task-technology fit (TTF) is another model that emphasizes individual impact [15]. According to TTF, individual impact refers to improved efficiency, effectiveness, and/or higher quality. Good fit between task and technology is assumed to increase the likelihood of utilization and also to increase the performance impact since the technology meets the task needs more closely [15]. TTF has also been applied in mobile work contexts, for example, in police field work [17], mobile electronic procurement [13] and insurance industry [28]. Although it is well understood that mobility and mobile context have remarkable effects on the use of mobile applications (e.g., [10], [49]), these aspects are seldom included in the models presented in this Section 2. As Gebauer and Tang [14] point out, user mobility and the context in which technology is being used need to be taken into account when applying TTF to mobile information systems. The context in which the mobile work is performed is challenging because of its dynamic nature. The battery life of the device, lighting, temperature, noise, or network connections are examples of the factors that may change while performing mobile work tasks. All these factors affect the users ability to perform work tasks with a mobile service. In order to have a comprehensive view of the aspects that affect the use of mobile business services in work context, we use the concept of mobile business service experience. In this paper, this

2. BACKGROUND
According to ISO 9241 standard [18], usability is the extent to which the intended users of a product achieve specified goals in an effective, efficient and satisfactory manner within a specified context of use. The usability of the service affects the user experience (UX), which is one of the key elements of a successful mobile service [43]. UX is a wider concept than usability but it cannot be easily defined [12]. User experience includes, for example, the tasks, the service provided, navigation, the design and the value the user achieves when using the service [43]. Furthermore, Roto argued that the main elements of UX are users internal state, context of use, and the actual system [37]. In our study, we state that identifying and evaluating impacts of user experience are vital for improvement, and therefore by minimizing cases where these impacts are incorrectly estimated, we can support the successful development of mobile business services. Many generic usability questionnaires have been developed for evaluating software systems, for example, the System Usability Scale (SUS) [2], the Software Usability Measurement Inventory (SUMI) [46], the Technology Acceptance Model (TAM) [6], the Questionnaire for User Interface Satisfaction (QUIS) [4], and the Post-Study System Usability Questionnaire (PSSUQ) [29]. In addition, some questionnaires have been deployed to evaluate the usability of specific software systems like web applications (WAMMI) [26] and electronic mobile products (Mobile Phone Usability Questionnaire (MPUQ)) [39]. The limitation of these questionnaires is that most of them were developed for a desktop environment and they do not cover the challenges in the usability

54

FP

concept describes the user experience of mobile business services including perceived usability of mobile business services, perceived fit for mobile working context and perceived impacts on mobile work productivity. In this way, mobile business service experience goes beyond the user experience towards understanding how employees and companies experience mobile business services. The dimensions of mobile business service experience are discussed in detail in the next section.

6. They should use their users terminology and their navigational structure should be organized in the way in which users think (User centeredness). 7. They should be adapted to each and every users own needs and capabilities (Personalization). Zhang and Adipat [51] studied current standards, handbooks and usability studies on mobile applications and identified nine generic attributes for usability of mobile applications: learnability, efficiency, memorability, errors, user satisfaction, effectiveness, simplicity (complexity), comprehensibility (readability) and learning performance. To study perceived usability of the mobile business service among users, the items described above, usefulness, efficiency, ease of use, user satisfaction, learnability etc., form the basis for these studies. Thus, the questionnaire developed should also include statements based on these items.

3. EVALUATING MOBILE BUSINESS SERVICE EXPERIENCE

The main purpose of this paper is to describe a questionnaire developed for evaluating mobile business service experience. From industry perspectives, when a mobile business service is being developed, a questionnaire is not the best possible method when developed services are still in their early stages (e.g., usability testing is a more valid tool then). In the later phases, a questionnaire as a research method is relatively efficient and cheap. The advantage of a usability questionnaire compared to laboratory evaluations, expert reviews and checklists is that a questionnaire is reasonable easy and quick to administer and also the real end users of the product are involved in the process. The measures made with a questionnaire can be used, for example, to compare user satisfaction between different versions of the same service or with competing services (benchmarking). Mobile business services are increasingly international, and with a questionnaire it is easy to contact people regardless of their physical location. On the other hand, the limitations of a questionnaire are related to subjective opinions, absence of personal contact, and relatively general level of analysis. Therefore, it is clear that questionnaires cannot be the only tool to use for experience development. Next, we describe the dimensions of mobile business service experience more extensively.

3.2 Perceived fit for mobile working context

The existing usability measures have mainly been developed for desktop computers. By contrast, the measures for mobile business services must take into account the different contexts of use, and integration with network, as well as more traditional working environments. Many studies have emphasized that the context in which mobile systems are used must be understood for designing successful mobile systems (e.g., [1], [10], [42], [49]). The term context is defined in various ways in the literature. The context often consists of the users, tasks, equipment, and the physical and social environments in which a product is used [18]. Mobile business services can be used with different devices, such as mobile phones or fixed in-car systems, and the context measures need to take into account the appropriateness of the device for the users task. For example, the nature of knowledge work or transportation is different and imposes different requirements for the device but also for the service. Work tasks may also be routine or non-routine [13]. In addition, the existing tools do not take into account the fact that the service is typically only part of the story the service may fail to support the work task because of problems in some other parts of the service. The mobile service rarely acts as an independent tool in the working environment but information needs to be transferred somewhere. Thus, it is important that the mobile service and the device enable fast and easy connection to other devices and users. The interaction between a user and a mobile system may fail if the characteristics of the context of use have not been fully understood. For example, interruptions and distractions [42] as well as screen and keyboard size [49] affect the ease of using a mobile system. Because of the limitations of the input capacity of the mobile device, text entry should be minimized [21] and lists and menus should be used. In addition, environment may have effects on the performance of mobile devices, for example, the capacity of batteries [16]. If a battery runs out or the network connections fail during the work task, the whole work task may fail and must be started again. Compatibility between a new mobile system and existing information systems must also be considered when mobile systems are designed [10].

3.1 Perceived usability of mobile business services

As stated earlier, usability is one of the essential success factors of mobile business services but usability itself is not a guarantee for success; the mobile worker has to experience perceived usability of the used service. A mobile business service needs to be effective, efficient, pleasant, and easy to use to make it easier for mobile workers to accept a new service in their daily work. In business use, severe errors should not occur even when the users are learning to use the service and the learning should not take too much time. Oinas-Kukkonen and Kurkela have defined seven key principles for highly goal-driven mobile services [33]: 1. Such services should provide information that users need when they are on the move (Mobility). 2. They should make the lives of their users easier (Usefulness). 3. They should include only relevant information (Relevance). 4. They should be simple and easy to use (Ease of use). 5. The most important information should be the easiest to locate (Fluency of navigation).

FP

55

3.3 Perceived impact on mobile work productivity

Productivity improvement is usually mentioned as one of the main success factors from the company point of view [35], [11]. In this paper, perceived impact on mobile work productivity refers to positive or negative impact of using mobile business services on the level of employee productivity. Employee productivity is a partial productivity measure that can be calculated by dividing the quantities of outputs by the quantities of inputs used (e.g. the number of working hours). Calculating work productivity, however, is challenging in situations when inputs or output are not comparable in either characteristics or quality [24]. Thus, subjective estimation is a good way to evaluate productivity, for example, in knowledge work and in services due to their intangible nature and varying quality of inputs and outputs. Companies can improve mobile work productivity in several ways, for example, by reducing or eliminating unproductive travel, improving data accuracy and avoiding unnecessary work by entering data only once while on location [35]. In addition, productivity may be improved, for example, by allowing mobile workers to remain productive in situations where they would otherwise experience down-time, including travel to and from work or client sites, and around the corporate campus [11]. Productivity impacts enabled by mobile business services are heavily dependent on the context of use. For example, mobile field workers benefit from the option to store data already in the field. Due to this, they have less paperwork and need less travel from and to the office during the working day [47]. On the other hand, mobile business travelers, for example consultants or sales force, need information access and real-time data in order to work more effectively in the dead time spent in transit or waiting [34]. In addition, mobile field workers can be tracked much more easily to allocate unscheduled work [47]. This in turn provides better resource management. On the other hand, business travelers can monitor the activities of remote colleagues to maintain awareness of what is going on in the office as well as to build a sense of community [34]. This may have an impact on work satisfaction.

established by literature reviews and interviews with managers, developers, usability specialists and customers of mobile business services. In our interviews, there were altogether 14 participants during spring 2006. All interviews were recorded and transcribed for future analysis. The interviews were analyzed qualitatively. In the following sections, the refinement work for the questionnaire will be described more specifically.

4.1 Initial Questionnaire development - Item generation

Items for MoBiS-Q were drawn from the literature, existing questionnaires and also from case studies with mobile business services. From earlier case studies, we first identified ten success factors of mobile business services. These success factors include ease of installation, learnability, ease of use, efficiency, effectiveness, user satisfaction, factors related to mobile context, safety, support, and mobile work productivity [30]. Most of these factors are included in the current tools for usability measurement. For example, effectiveness, efficiency, and user satisfaction are emphasized in the ISO standard definition of usability [18]; efficiency, learnability, and satisfaction are emphasized by Nielsen [32], and ease of use is, for example, part of TAM [6]. In addition, installation and support (technical manuals and on-line help) are part of QUIS [4]. Thus, the existing usability questionnaires were used as the basis for the present tool, although they needed to be revised for present purposes because they do not specifically focus on the issues relevant for the mobile work context. Thus, we revised the otherwise relevant questions from the existing tools to better measure the success factors that are important for mobile business services. In addition, new dimensions needed to be added to capture all the important factors in relation to the mobile business service experience, such as the issues related to the mobile device, environment and the overall system, as well as safety and mobile work productivity. After identifying these ten dimensions, we created the first version of the questionnaire. The items from the existing questionnaires were selected based on their relevance to mobile work context. The items selected were modified so that mobile service and work tasks were emphasized. Furthermore, new items were created for each success factor. After this, three researchers analyzed each success factor and grouped the items under three broader dimensions identified from the literature and interviews: usability, mobile working context, and mobile work productivity. The first dimension, usability of mobile business service, includes 33 items. If the mobile business service is easy to adopt and accepted as a part of users work, the overall work satisfaction and workers productivity are assumed to increase. In a business context, workers often do not have a choice whether to use or not the mobile business service and they often do not even have a say when the company is deciding which service to introduce. Thus perceived usability, subjective satisfaction and ease-of-use are extremely important factors as they have a major effect on productivity. The dimension of mobile working context includes 24 items related to the actual physical context, i.e. mobile device and

4. QUESTIONNAIRE DEVELOPMENT METHODS

In order to develop the mobile business service questionnaire (MoBiS-Q) for measuring mobile business service experience, we iteratively designed and refined items for measuring the three dimensions described in the previous section, namely usability, mobile working context, and mobile work productivity. To ensure that our items evaluate what they are intended to, we pre-tested MoBiS-Q and used it in three pilot tests. We then compared the results to find out whether the items were suitable for measuring the success of a certain mobile business service. The goal of this process was to create reliable and valid items. The primary challenges we had were on examining to what extent the evaluation tool fulfils the content validity (do items evaluate what are intended to) and construct validity (does our evaluation tool reflect the construct of a successful mobile service), and furthermore, how well we could generalize our result. The content and construct validity of the dimensions were tentatively

56

FP

environment, as well as the overall system context (see, e.g., [7]). How a mobile business service suits a mobile working context is measured in our questionnaire by four dimensions: a mobile device (e.g., screen size, battery capacity, using with one hand), environmental effects (e.g., work safety), a mobile service and work tasks. All these dimensions include a view of mobility; using a mobile service while moving. In order to more clearly identify the reasons behind the problems, we separated a mobile device from a mobile service. Finally, in order to evaluate impacts on mobile work productivity, we added 21 items. The items related to efficiency, effectiveness and usefulness are especially closely related to productivity.

4.2 Iterative item refinement

The preliminary item set was then further refined and assessed through pre-testing and pilot tests. First, we had altogether 78 items that three researchers analyzed for their redundancy and relevancy. On the basis of this analysis, 33 items were removed and the questionnaire was formed containing 12 items for perceived usability, 11 items for perceived fit for mobile working context and 9 items for mobile work productivity. In addition, due to the small amount of standardized productivity related questionnaires, 11 items were left for testing various aspects of productivity impacts. Respondents had the option to answer does not apply to my work tasks to these items. For the questionnaire, items were formulated as statements with a 7 point Likert scale from 1 = Strongly disagree to 7 = Strongly agree. In addition, there was also an option Dont know.

in Finland. The mobile business service includes mobile work email, calendar, and employee directory. The mobile business service was being piloted at the time of the study. The sample size in this pilot study was 163. Respondents were selected by the case company using systematic sampling, so that those who were really using the service were chosen for the study. There were altogether 84 responds in this study and the response rate was 51 %. The data was gathered using a web-based survey tool. Respondents were between 27 and 60 years old. The average age of the respondents was 40 years. Most of the respondents were men (73 %). The users used the service daily. Over 60 % of the respondents had not used a similar kind of service earlier. Over 70 % of those with previous experience claimed that the previous experience had been beneficial when learning to use the service. Almost every respondent (98 %) used the mobile service via a mobile phone. The third pilot test, conducted together with a company in the area of mobile service creation, also included knowledge workers located all over the world. The solution evaluated was a combination of push email, calendar, contacts, and other mobile services, integrated into one user interface. The sample size was 100 participants. The data was gathered using a web-based survey tool. The sample size was one hundred and the response rate was 34 %. The average age of the participants was 36 years (range 2647 years). The majority of the respondents were male (86 %). Of the participants, 80 % had used a similar kind of mobile service earlier and over 77 % used the particular mobile business service evaluated more than five times per day. Their average experience of using this particular mobile business service was 26 weeks. Table 1 Summary of the pilot tests Pilot test #1 (n = 48) sample size response rate Age (mean) Gender Men Women N/A 40 5 3 60 22 2 30 2 2 100 48 % 46 Pilot test #2 (n = 84) 163 51 % 40 Pilot test #3 (n = 34) 100 34 % 36

4.2.1 Pre-testing
The questionnaire was pre-tested with four people. They completed a paper questionnaire while a researcher was observing them. The test users thought aloud as they completed the questionnaire. The questionnaire was modified with the results of the pre-tests. For example, the layout of the questionnaire form became clearer when each of the dimensions was put on separate pages. In addition, the scale was added at the beginning of every part of the questionnaire and the scale was described both numerically and verbally. Some items were also rephrased to be more unambiguous.

4.2.2 Pilot tests

After pre-testing, we conducted three pilot tests of the questionnaire, which are summarized in Table 1. The first pilot test involved taxi drivers using a taxi order booking and dispatching service in their work. This study was conducted together with a service developer and a local area taxi company in Finland. The sample size was one hundred and the response rate was 48 %. The sample was chosen by using systematic sampling, so those who had used the taxi service at least one month were chosen from the user population. The data was gathered by paper questionnaire. The average age of all the respondents was 46 years. The respondents had used the taxi booking service approximately two years. Most of the respondents were male (over 80 %). One fifth of the respondents had used a similar taxi booking service before. A third of them thought the earlier user experience was useful for learning to use the new service. The second pilot test was conducted with a telecom company providing mobile business services for their knowledge workers

The total number of items was between 43 and 46 in the three pilot tests (Table 2). The first and second pilot tests included otherwise the same items but one item related to productivity (quality of output) was added to the second pilot test. This item was also included in the third pilot test. The third pilot test was integrated into the realistic ongoing mobile business service development process and the participants of our study were the same end-users who evaluated the service in the realistic development process as well. Because of the nature of the service development process, the end-users installed the service into their mobile phones by themselves. Therefore, three questions related to the installation phase were added at the beginning of the questionnaire. In addition, one item related to the mobile working context was removed from the third pilot test because of its redundancy with another item. In the third pilot test, we combined

FP

57

and reformulated four items into two and added two new items related to general functions that are useful for mobile workers [50]: information processing and mobile communication. In addition to these items, we asked how often and what problems users had experienced with the mobile business service related to using mobile business service, missing or wrong information, mobile device, network connection and environmental effects. There were also some open ended questions and questions related to the background information of the respondents for statistics. Table 2 shows the degrees of internal reliability, coefficient alpha (Cronbachs ) for all dimensions and the number of items in the pilot tests. Table 2 Cronbachs for pilot tests (# of items per dimension) Pilot test #1 (n = 48) Usability Mobile working context Mobile work productivity 0. 84 (12) 0.69 (11) Pilot test #2 (n = 84) 0.68 (12) 0.58 (11) Pilot test #3 (n = 35) 0.82 (15) 0.64 (10)

11. Ease of navigation 12. Efforts made when starting to use 13. Functions are simple to use 14. Concentration 15. Recommend to others Fit for mobile working context (16-25) 16. Ease of use with a device 17. Screen size 18. Battery capacity 19. Suitability of a device for working on the move 20. Information input 21. Using a device with one hand 22. Environmental context (coldness, sunshine, darkness, noisiness, etc.) 23. Safety risks while working on the move 24. Ease of use while on the move 25. Ease of use in a hurry

0.95 (9) 0.96 (11)

0.91 (10) 0.94 (11)

0.88 (10) 0.92 (11)

Most of the values of coefficient alpha exceeded 0.80, which is very good level of alpha [9]. The alphas related to the mobile work context were lower than the others and even at an unacceptable level ( < 0.60) [9]. Deleting some items (e.g., the item related to the environmental context) would increase the alpha of working mobile context to the level of 0.6 in all the cases (pilot tests #1, #2, #3). In the case of the pilot test #2 the alpha of usability is under 0.7 but if two of the items (related to errors and concentration) are deleted the alpha increases to over 0.7.

Perceived impact on mobile work productivity (Part A: 26-35) (Part B: 36-46)

26. Satisfaction with efficiency at work 27. Use accelerates work performance 28. Use improves work motivation 29. Use improves work satisfaction 30. Use improves fluidity of work 31. Accomplishing work task effectively 32. Use increases productivity 33. Able to perform tasks in less time than before 34. Able to complete tasks easier 35. Satisfaction with the quality of work tasks 36. Less working phases 37. Less time to go through working phases 38. Less additional traveling 39. Better access to information needed at work 40. Decision making 41. Distributing work tasks 42. Planning and coordinating work tasks 43. Gathering information 44. Sharing information 45. Information processing 46. Mobile communications

5. RESULTS
As a result of iterative item generation, item refinement and pilot tests, MoBiS-Q, a questionnaire for measuring mobile business service experience was developed. Three dimensions of the questionnaire and items related to each dimension are presented in Table 3. Table 3 Dimensions and items of MoBiS-Q Dimension Perceived usability of a mobile business service (1-15) Items 1. Easy to install and setup to the device 2. No errors in the installation process 3. Speed of installation 4. Easy to learn 5. Easy to become skilful 6. Reliability 7. Suitability for work tasks on the move 8. Errors 9. Quick enough 10. Functions are necessary

After the development of the MoBiS-Q, three researchers also discussed the relevance of each item based on the responses. Most of the items related to the perceived usability of a mobile business service were seen as relevant to all pilot tests. For example, in all

58

FP

of the three cases, parallel results were gained for statements related to learnability, suitability for mobile work tasks and necessity of the functions in mobile service. Moreover, the results showed that all users were ready to recommend the mobile service to others doing similar kind of work. However, differences were found between the responses of knowledge workers and taxi drivers. For example, the statements related to the reliability of a mobile service, amount of attention needed to use a mobile service and simplicity of use seemed to be more critical factors for taxi drivers than for knowledge workers. This is reasonable; while taxi drivers have to operate the mobile business service while on the move (driving) whereas knowledge workers can use their service, for example, while sitting on the train, bus or in meetings. There were also some differences among the knowledge workers in results from pilot tests #2 and #3. For example, responses to statements related to faultlessness and reliability were not in line in these two tests. From these results, one could come to the conclusion that one service could have had some problems in its functionality. Some of the items related to perceived fit for a mobile working context had some differences among the responses. For example, one group of knowledge workers (the pilot test #3) thought that the battery capacity of their mobile device was not sufficient for using their mobile service. Lengthy use of a mobile service via mobile phone drains the power remarkably. However, we found that the battery capacity of the mobile device is not a relevant question for users whose mobile device is an embedded system, like taxi drivers in-car systems. In addition, using a mobile device with one hand was rated differently. The taxi drivers were more satisfied with using their mobile service with one hand whereas the knowledge workers were not so satisfied. The reason for this may be that the taxi drivers device is a touch screen and the size of the screen is larger, whereas knowledge workers were mainly using their mobile phones. For the same reasons as before, mobile work safety seemed to be a more relevant item for taxi drivers than for knowledge workers. On the other hand, the data security could be a more valuable item to be added for knowledge workers. It would be beneficial to study how data security is experienced, especially how much the users are ready to suffer without decreasing the usability of a mobile service. Sometimes data security may be cumbersome from a users point of view, for example, because of passwords or duration of sessions. Therefore, it is important for service developers to find the balance between data security and usability of a mobile service. In addition to these differences, we found that some terms are suitable for some people and some are not. For example, the terms working context, mobile work context, or context of use have different meanings for knowledge workers and taxi drivers. Working context for knowledge workers means, for example, their desktop and devices. When they send e-mail in a railway station, they do not think the place is their working context. For taxi drivers the meaning of the working context is wider. They think that the environment (e.g., a car and traffic) where they move and perform their work tasks is the working context. Related to the context of use and the complexity of mobile services, it may be difficult for the users to know why there are problems. The users do not know whether the problems are

caused by the device, the service or network connections. However, this is very important for service providers to recognize if the problem is in the service or in the device. On the other hand, it is essential to take into account that the service and the device are compatible. Environmental effects, like sunshine and cold weather, may make the use of the mobile service difficult. Therefore, it is important to cater for these when designing mobile systems for users whose context of use is not alterable. For example, taxi drivers use their mobile system while driving a car. They cannot choose the context of use and minimize the negative effects of environment. On the other hand, knowledge workers do not use their mobile service only while walking or driving. They can move to another place or indoors where the interaction with a mobile service may be better. In this way, they can minimize the environmental effects. After these dimensions, there were first ten more general productivity items related to perceived impacts on mobile work productivity (Part A). One of the items asked directly if the use of a mobile business service increases work productivity. The other statements assessed factors closely related to productivity, i.e. how satisfied the respondents were with their efficiency at work, if the use of a mobile business service was experienced as accelerating their work performance or improving their work motivation or satisfaction. In addition, there were statements asking if the use of mobile business services improves fluidity of work or enables workers to accomplish work tasks efficiently, quicker, or easier. In addition, quality of the outputs the work tasks performed was also one item assessing productivity. These items were relevant in all of the pilot tests due to their general nature. In order to understand the relevance of more specific productivity statements, the respondents could answer Does not apply to my work/ the mobile service to the eleven additional productivity items (Part B). These items were named as positive impacts on mobile work productivity. Two out of eleven items seemed to be irrelevant for knowledge workers in the second and third pilot tests: There are fewer working phases to complete now and I need less time to go through different working phases. These were, however, important for taxi drivers. Items related to availability of information, decision-making and whether gathering or sharing information has improved seemed to be relevant in all of the pilot tests. In the third pilot test, items related to information processing and mobile communication were highly relevant for knowledge workers. Some differences in the responses may be due to the nature of the work tasks. For example, taxi drivers work is routine, whereas knowledge workers use their mobile service, for example, when they have down time while traveling or between meetings. Taxi drivers also perform almost all their work tasks with the mobile service whereas knowledge workers perform only some supportive tasks with their mobile service, for example, reading and responding to emails. Thus, the estimation of productivity impact may not be comparable between such a different work tasks.

5.1 Process development

In addition to the development of a questionnaire, the use of MoBiS-Q was evaluated in the actual development processes of

FP

59

mobile business services (the three pilot tests described earlier). We found that fast, real time, continuous and close feedback is one of the keys for successful service development cycle. The benefits that a questionnaire approach can bring are related to efficiency and scale. To add value, the tools in use need to be scalable to the needs of design and development processes [38], [27]. This requires continuous readiness for analysis and synthesis. Focus on sufficiently generic questions, with the option for open feedback, can make it easier to get continuous, fast feedback. We found that this type of analyzed and summarized information is relevant, and also needed for service development. However, focus on summaries only can make the view too narrow. Details can bring the needed added value for developing the experience further. For example, most of the respondents in this study experienced that the network connection is often or quite often the reason for bad mobile business service experience. This is a known problem, but also something that no technology or service provider can solve alone. If we focus on more system details, it is also important to remember that according to the feedback received in this study, a 15 minute questionnaire can be too long for people who need to respond to questionnaires on a continuous basis. Due to the complex nature of mobile business service, a questionnaire was experienced to be more beneficial when more than only one aspect of the service is combined in the contents. As MoBiS-Q combines usability, context and productivity factors together, the interest group for the results is wider, including not only designers and developers but also managers, sales people and customers. Therefore, co-development of questionnaire content is also needed. In any case, the responsibility for leading the planning activity needs to be clear: concentration needs to be on listening to core messages of people participating in design and development activities [38], [23], [40]. Important factors for the evaluation of business services include the way of presenting the results: the simpler the presentation of the results the easier it is to communicate the findings in the organization. Therefore, graphics is a recommended format for presenting the results, and also, if there is a need for a quick and short reporting, a single average figure or grade can be calculated and used to communicate the measured experience of a service. When the findings are presented in a simple way and the questionnaire is kept more or less the same, the results are comparable with findings based on competitors services, with different versions of a service, or even with other mobile business services. Also, if the results are easy to present, they can be used in the sales process of a newly developed service.

of 0.70 (Table 2), some items need to be rephrased or deleted. The length of the questionnaire should also be considered. Due to the speed of mobile service development, analysis, synthesis, results generation and use need to be supported by the tool in use. From the industry perspective, the two most integral and demanding phases in tool development are on the one hand the development of a dynamic but exact data algorithm, and on the other hand mapping dynamic functionality with visualization. The role of visualization is to make the cognitive processes for decision-making faster. Interaction can add flexibility, or more detail in the faster decision-making process [3], [45]. In this study, the MoBiS-Q tool does not yet contain any automatic or visual functionality for analysis or synthesis. Because of this, the time used for these phases is too long from the service design and development phase point of view, and the cognitive process for making decisions based on the synthesis, continues to be slow. Also, due to relatively small response rates, no actual statistical analysis could be used here. In addition, there needs to be easy access to the tool for any of the people belonging to the network of users, and their access needs to be to real time, analyzed data. On the other hand, security needs to be taken into account as well. Keeping security in mind, integrating the knowledge from this study with some already available, open source based statistical visualization tool could be the next iterative step towards a competitive solution [36], [41].

6. CONCLUSIONS
The primary contributions of this paper were the definition of three dimensions associated with the mobile business service experience and the development of new measurement tool, MoBiS-Q, for measuring these dimensions. These dimensions include perceived usability of mobile business services, perceived fit for mobile working context and perceived impacts on mobile work productivity. When evaluating the usability and productivity issues together, the idea was to understand the critical factors in the mobile work context. As User-Centered Design approaches recommend, it is crucial that the development of a new service is a joint operation between relevant parties and departments in an organization, including user representatives, sales, marketing, product management, technology, and usability in the process. Therefore, the MoBiS-Q questionnaire preparation and analysis can be seen as an ideal tool for bringing together all those parties to work together during the development process, composing the set of questions and evaluating the results together. The study adds value by showing that more focus on the system context related details could be the way to develop the service experience further. Cross organizational cooperation agreements and activities between different service and device providers can make the services more integrated in the context, and more appealing for the user. The benefits of the cooperationcompetition dilemma can also bring value in problem solving, and support channels. For the user it is difficult to know where the actual problem lies, but if not solved, the service experience may be bad, and the productivity of an employee may be lower. The most difficult challenges (e.g., reliability of networks while on the move) cannot be solved by any service or technology provider alone.

5.2 Development of the evaluation tool

MoBiS-Q provides a good starting point for future discussions related to mobile business service experience. Combining users subjective experiences of a service and the productivity requirements of a company in a mobile work context, it brings together various parties involved in developing and deploying mobile business services. MoBiS-Q still needs some refinement and testing with a larger sample in order to be better applicable in different industries. In addition, as not all items related to the mobile working context did passed the minimally acceptable level

60

FP

Due to complexity of the system solutions, and also due to many active roles in the service network, the planning phase of the evaluation process will need some more attention, and it needs to be cooperative. Responsibilities need to be clear. Other tools will be needed, too; questionnaires cannot say it all. With this study, the cross organizational planning approach was not yet in use, therefore there were some challenges in content generation. In addition to planning, analysis, synthesis and result generation will also need more support from the MoBiS-Q tool. The tool will need to contain more automatic, statistical basic methods with automatic visualization. The idea presented here would be to enable the integration of this tool with some already available, possibly open source based, statistical visualization tool. The tool needs to be easily accessible, but access control is essential. The requirements presented here are basic requirements for wider adaptation of the tool. The process measuring the final service experience needs to be faster, more flexible and more visual.

Year Update. Journal of Management Information Systems, 19, 4, 9-30. [9] DeVellis, R.F. 2003. Scale Development Theory and Applications. 2nd edition. Sage Publications. [10] Dix, A., Rodden, T., Davies, N., Trevor, J., Friday, A., and Plafreyman, K. 2000. Exploiting Space and Location as a Design Framework for Interactive Mobile Systems. ACM Transactions on Computer-Human Interaction, 7, 3, 285321. [11] Evans, N.D. 2002. Business Agility Strategies for Gaining Competitive Advantage through Mobile Business Solutions. Upper Saddle River (N.J.): Prentice Hall. [12] Forlizzi J., and Battarbee K. 2004. Aesthetics, ephemerality and experience: Understanding experience in interactive systems. In proceedings of Designing interactive systems: processes, practices, methods, and techniques, ACM Press, 261-268. [13] Gebauer, J., and Shaw, M.J. 2004. Success Factors and Impacts of Mobile Business Applications: Results from a Mobile e-Procurement Study. International Journal of Electronic Commerce, 8, 3, 19-41. [14] Gebauer, J., and Tang, Y. 2007. Applying the Theory of Task-Technology Fit to Mobile Information Systems: The Role of User Mobility. Proceedings of the International Conference on Mobile Business. [15] Goodhue, D. L., and Thompson, R. L. 1995. Tasktechnology fit and individual performance. MIS Quarterly, 19, 213-236. [16] Gorlenko, L. and Merrick, R. 2003. No wires attached: Usability challenges in the connected mobile world, IBM Systems Journal archive, 42, 4, 639-651 [17] Ioimo, R.E. and Aronson, J.E. 2004. Police field mobile computing: Applying the theory of task-technology fit. Police Quarterly, 7, 4, 403-428. [18] ISO 9241-11. Ergonomic requirements for office work with visual display terminals part 11: guidance on usability (ISO/IEC 9421-11: 1998). [19] Kaasinen, E. 2005. User Acceptance of Mobile Services Value, Ease of Use, Trust and Ease of Adoption. VTT Information Technology, Helsinki. [20] Kakihara, M., Sorensen, C. 2004. Practicing Mobile Professional Work. Tales of Locational, Operational and Interactional Mobility. INOFO: The Journal of Policy, Regulation and Strategy for Telecommuniacation, Information and Media, 6, 3, 180-187. [21] Kalakota, R., and Robinson, M. 2001. M-Business The Race to Mobility. McGraw-Hill. [22] Karahanna, E., Agarwal, R., and Angst C. 2006. Reconceptualizing Compatibility Beliefs in Technology Acceptance. MIS Quarterly, 30, 4, 781-804. [23] Kautonen, M, Schienstock, G. and Tiainen, M. 1999. Knowledge-Intensive Business Services Their Role and Development in the Tampere Urban region. In: Schienstock, Gerd and Kuusi, Osmo (eds.): Transformation towards a Learning Economy. The Challenge for the Finnish

7. ACKNOWLEDGMENTS
This study was conducted as part of the MOMENTO project (Ensuring the success of mobile services for business usage by multidisciplinary measurement tools). MOMENTO is a research and industrial cooperation project that started at the beginning of 2006. The project is financed by Tekes (the Finnish Funding Agency for Technology and Innovation) and five companies. We would like to express our gratitude to the representatives of the case companies and their customers who participated in this study.

8. REFERENCES
[1] Barnard, L., Yi, J., Jacko, J., and Sears, A. 2007. Capturing the effects of context on human performance in mobile computing systems. Personal and Ubiquitous Computing, 11, 2, 81-96. [2] Brooke, J. SUS A quick and dirty usability scale. Available online at http://www.usability.serco.com/trump/documents/Suschapt.d oc. [3] Card, Stuart K., MacKinlay, J. and Schneiderman, B. 1999. Information Visualization: Using vision to think. Academic press. [4] Chin, J.P., Diehl, V.A., and Norman, K.L. 1988. Development of an instrument measuring user satisfaction of the human computer interface. Proceedings of CHI88, 213218. [5] Corso, M., Giacobbe, A., Martini, A., and Pellegrini, L. 2006. What Knowledge Management for Mobile Workers? Knowledge and Process Management, 13, 3, 206-217. [6] Davis, F.D. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, September, 318-340. [7] Delone, W.H., and McLean, E. R. 1992. Information systems success: The quest for the dependent variable. Information Systems Research, 3, 6095. [8] DeLone, W.H., and McLean, E.R. 2003. The DeLone and McLean Model of Information Systems Success: A Ten-

FP

61

Innovation System. Finnish National Fund for Research and Development. Helsinki. [24] Kemppil, S. and Lnnqvist, A. 2003. Subjective Productivity Measurement. The Journal of American Academy of Business, 2, 2, 531-537. [25] Kim, H-W., Chan, H.C., and Gupta, S. 2007. Value-based Adoption of Mobile Internet: An empirical investigation. Decision Support Systems, 43, 111-126. [26] Kirakowski, J., & Cierlik, B. 1998. Measuring the usability of web sites. In Proceeding of Human Factors and Ergonomics Society 42nd Annual Meeting, Santa Monica, CA. [27] Larman, G. 2004. Agile & iterative development - A manager's guide. Addison Wesley. [28] Lee, C-C., Cheng, H.K., and Cheng, H-H. 2007. An empirical study of mobile commerce in insurance industry: Task-technology fit and individual differences. Decision Support Systems, 43, 95-110. [29] Lewis, J.R. 1995. IBM Computer usability satisfaction questionnaires: psychometric evaluation and instructions of use. International Journal of Human-Computer Interaction, 7, 1, 57-78. [30] Markova, M., Aula, A., Kulju, M., Vainio, T. and Wigelius, H. 2007. MoBiS-Q - A tool for evaluating the success of mobile business services. In Proceedings of MobileHCI 2007, 238-245. [31] Markova, M., Aula, A., Lnnqvist, A. and Wigelius, H. 2008. Identifying and measuring the success factors of mobile business services, International Journal of Knowledge Management Studies, 2, 1, 4963. [32] Nielsen, J. 1993. Usability engineering. Academic Press Inc.,Boston. [33] Oinas-Kukkonen, H. and Kurkela, V. 2003. Developing Successful Mobile Applications. International Conference on Computer Science and Technology (IASTED), 50-54. [34] Perry, M., OHara, K., Sellen, A., Brown, B., & Harper, R. 2001. Dealing with Mobility: Understanding Access Anytime, Anywhere. ACM Transactions on ComputerHuman Interaction, 8, 4, 323-347. [35] Rangone, A., and Renga, F. 2006. B2e mobile internet: an exploratory study of Italian applications. Business Process Management Journal, 12, 3, 330-343. [36] Rhyne, T.-M. 2005. Visualization viewpoints. http://ieeexplore.ieee.org/iel5/38/35448/01683688.pdf?tp=&i snumber=&arnumber=1683688. [37] Roto, V. 2006. Web Browsing on Mobile Phones Characteristics of User Experience. Doctoral dissertation,

TKK Dissertations 49, Helsinki University of Technology, Finland. [38] Rosenbauhm S., Rohn J. A. and Humburg J. 2000. A Toolkit for Strategic Usability: Results from Workshops, Panels & Surveys. CHI 2000 Proceedings. ACM, Inc. [39] Ryu, Y.S., and Smith-Jackon, T.L. 2006. Reliability and Validity of the Mobile Phone Usability Questionnaire (MPUQ). Journal of Usability Studies, 2, 1, 39-53. [40] Saint-Onge, H. and Wallace, D. 2003. Communities of Practice for Strategic Butterworth & Heineman. Leveraging Advantage.

[41] Sasse, M.A., Flechais I. 2005. Usable security. In: Cranor F. and Garfinkel S. Designing secure systems that people can use. OReilly. [42] Savio, N. and Braiterman, J. 2007. Design Sketch: The Context of Mobile Interaction. In Proceedings of MobileHCI 2007, 284-286. [43] Schonberg, E., Cofino, T., Hoch, R., Podlaseck, M., and Spraragen. N, S. L., 2000, Measuring success. Commun. ACM, 43, 53-57. [44] Sheng, H., Nah, F.F., and Siau, K. 2005. Strategic implications of mobile technology: A case study using Value-Focused Thinking. Journal of Strategic Information Systems, 14, 269-290. [45] Siirtola H. and Mkinen, E. 2005. Constructing and reconstructing the reorderable matrix. Information Visualization 4, 1, 32-48. [46] Software Usability Measurement Inventory. Information on line at http://www.ucc.ie/hfrg/questionnaires/sumi/. [47] Verburg, R.M., Testa, S., Hyrkknen, U. and Johansson, N. 2006. Case descriptions of mobile virtual work in practice. In Andriessen, J.H.E. and Vartiainen, M. (Eds.) Mobile virtual work, Springer, Heidelberg, 267-288. [48] Wu, J-H., Wang, S-C. 2005. What drives mobile commerce? An empirical evaluation of the revised technology acceptance model. Information & Management, 42, 719-729. [49] York, J., Pendharkar, P.C. 2004. Human-computer interaction issues for mobile computing in a variable work context. International Journal of Human-Computer Studies, 60, 5-6, 771-797. [50] Yuan, Y., Zheng, W. 2006. The fit between Mobile Task and Mobile Work Support: A Theoretical Framework. Proceedings of the International Conference on Mobile Business. [51] Zhang, D., Adipat, B. 2005. Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications. International Journal of Human-Computer Interaction, 18, 3, 293-308.

62

FP