Professional Documents
Culture Documents
1, 2011 1
Dae-Ho Byun*
School of Economics and Logistics,
College of Economics and Commerce,
Kyungsung University,
Busan 608-736, Republic of Korea
E-mail: dhbyun@ks.ac.kr
*Corresponding author
Gavin Finnie
School of Information Technology,
Bond University,
QLD 4229, Australia
E-mail: gfinnie@bond.edu.au
Reference to this paper should be made as follows: Byun, D-H. and Finnie, G.
(2011) Evaluating usability, user satisfaction and intention to revisit for
successful e-government websites, Electronic Government, An International
Journal, Vol. 8, No. 1, pp.119.
Information Systems teaching and research for over 30 years. He has published
over 100 papers in journals and refereed conferences as well as a book and
several book chapters. His research interests are in the area of AI/expert system
applications in information systems, intelligent agents, electronic business,
business process management and real-time business intelligence.
1 Introduction
usability, user satisfaction and intention to revisit? Finding these relationships provides us
with guidance for improving e-government websites and they can be used as a model for
promoting e-government website success.
The objective of this paper is to describe the importance of usability in evaluating
websites. We find major factors affecting usability of e-government websites and show
causal effects to determine the extent to which usable websites affect user satisfaction
and intention to revisit. Our paper is part of research outcomes published in Korean
journals by the first author (Byun, 2005, 2007). We performed user testing on the
websites of representative administration departments in Korea, which are linked by the
Korean e-government web portal. Users were requested to answer the usability evaluation
checklists after finding correct answers to required questions by navigation. The target
website of the Korean government portal, which opened in 2002, had a top rank in 2006
and 2007 in the e-government country ranking (West, 2007).
The paper is organised as follows: First, a research model and hypotheses for
the research questions are developed from recent research on e-government focused
on e-government success, efficiency and evaluation. Second, data is collected by
user testing, analysed by factor analysis and a structural equation model developed.
Third, we suggest findings that include ranking of e-government websites, factors
affecting usability and causal effects between usability, user satisfaction and intention to
revisit.
2 Literature review
The constructs we will consider are perceived usability (PU), user satisfaction (US) and
intention to revisit (IR). Perceived usability means the usability based on the users
viewpoint. Our objective is to find a causal relationship between these three constructs
such that usable e-government websites affect user satisfaction and intention to revisit
sites, respectively. The basic proposition of our research model is that user satisfaction is
determined by perceived usability. Similarly, intention to revisit is also influenced by
user satisfaction, i.e., the more usable the website, the higher the user satisfaction, and the
more the user satisfaction, the stronger the intention to revisit the website.
Flavin et al. (2006) showed that greater usability of websites was found to have
a positive influence on user satisfaction and this also generated greater website loyalty.
We can assume loyalty is the same concept as intention to revisit. Floropoulos et al.
(2010) developed a model that included the constructs of service quality, perceived
usefulness and user satisfaction for government information system success. The results
provided evidence that there were strong connections between these constructs. We can
define usefulness as subset of usability. Oztekin et al. (2009) proposed a methodology to
Evaluating usability, user satisfaction and intention 5
combine web-based service quality and usability dimensions of information systems and
revealed a strong relationship between quality and usability.
From the preceding discussion, we can define the relationship between usability and
user satisfaction. However, we divided user satisfaction into two constructs. One is
physical satisfaction (PS). Although PS is a difficult concept to define precisely, for this
research we consider it to include the view that users are not fatigued while performing
the task, are happy to proceed with the task and are willing to continue with the next task.
The other is achievement satisfaction (AS), measuring whether users achieved their
objectives well on the website. PS occurs in the process of finding the information users
want, but AS occurs after users find the required information. We hypothesise:
H1: Perceived usability positively affects physical satisfaction.
H2: Perceived usability positively affects achievement satisfaction.
Chen and Macredie (2005) determined that usability for e-shopping interfaces was
critical to help users to obtain their desired result. Casal et al. (2008) confirmed the
influence of website usability on consumer satisfaction and showed that usability played
a special role in the loyalty formation process. It follows that
H3: Perceived usability positively affects intention to revisit.
Teo et al. (20082009) found that quality perceptions of citizens were affected by their
trust in e-government websites and intention to continue using was affected by user
satisfaction. Deng et al. (2010) showed that trust, customer satisfaction and switching
cost directly affected customer loyalty in using mobile instant messaging. From the
preceding discussion, we hypothesise:
H4: Physical satisfaction positively affects achievement satisfaction.
H5: Achievement satisfaction positively affects intention to revisit.
In the research model as shown in Figure 1, perceived usability is considered a prior
variable; user satisfaction is considered an intermediate variable; revisiting intention is
considered an achievement variable.
Items for user satisfaction were derived from the Spool et al. (1999) research.
User satisfaction was measured by seven items, which were physical fatigue (US1),
confusion during the task (US2), degree of stress after finding a correct answer (US3),
overall physical feeling (US4), actual speed of tasks (US5), satisfaction about the quality
of information provided (US6), and attitude about proceeding to another task after
completing a task (US7). Intention to revisit was measured by three items, which were
acquisition of information (IR1), civil appeal (IR2) and getting several documents on
the e-government websites (IR3).
Figure 1 gives a graphical representation of the above-mentioned hypotheses. These
hypotheses were tested with a questionnaire during user testing.
4 Method
5 Findings
In page design, item I-3 had the highest score and I-2 the lowest score. The subjects had
comparatively high satisfaction with navigation, but the sites failed to provide
convenience with proper mixing of graphic and text. In contents design, item II-5 had the
highest score, but II-10 had the lowest score. The subjects have comparatively
high satisfaction with the ease of reading the text, but the sites do not properly
10 D-H. Byun and G. Finnie
use video images. From the results, the websites were reasonably good at offering
information in general text mode, but were insufficient in providing information in
various modes like video and animation.
Site design was less than the average score of overall websites. Item III-4 had
the highest score and III-12 had the lowest score. This indicated that users had no
difficulty in obtaining information by navigation, but the websites thought only of
providing information one-way rather than aiming at the medium of mutual
communication.
Applying a t-test, all p-values of the five hypotheses were greater than the significance
level of 5%. We can conclude that internet skills and user experience have no effect on
usability and the websites have no differences in usability according to gender, internet
experience and internet usage.
We tested whether there was significant correlation among the ranks of page,
contents and site design. That is, we tested whether websites of good page design provide
good contents design, or site design. Table 4 shows Spearman rank-order correlation
coefficients. In general, two constructs are highly correlated when the correlation
coefficient is less than or equal to 0.60, or greater than or equal to 0.60. The ranks of the
page and content design construct were relatively highly correlated with 0.582 at
p-values of 0.018. Websites with a high rank in page design also had a high rank in
contents design. The ranks of the page and site design construct were highly
Evaluating usability, user satisfaction and intention 11
correlated with 0.632 at p-values of 0.009. The correlation coefficient of the contents and
site design construct was 0.774.
We can conclude that when the usability of contents design is high, the site design tends
to a high value. Because the rank orders of page, contents and site design showed
relatively high correlation, the hypothesis There were no differences of website ranks
by construct was accepted at the significance level 5%. We can conclude that websites
with high usability in one construct, therefore, showed high usability in the other
constructs.
The hypotheses H1H4 were all supported with t-values greater than the critical value of
1.96, but H5 was rejected at the 5% significance level (see Table 8). A significant
positive relationship was found between perceived usability and user satisfaction. This
implies that the higher the perceived usability, the more the user satisfaction. Similarly, a
significantly positive relationship was also found between PS and AS. This implies that
the higher the PS, the higher is the AS. In the structural equation model, we can say
that there is an indirect effect between two variables when the second latent variable
is connected to the first latent variable through one or more other latent variables.
Therefore, the perceived usability had an indirect effect on AS through PS. This suggests
that PS is a prior variable affecting the relationship between perceived usability
and AS.
As can be observed from Figure 2, the direct path between perceived usability and
intention to revisit was significant. This suggests that the higher the perceived usability,
the more the intention to revisit. This is an interesting result, i.e., to increase revisiting
Evaluating usability, user satisfaction and intention 15
The total effect between two latent variables is the sum of any direct effect and all
indirect effects that connect them. Quantitative analysis of the effects of each construct
on intention to revisit reveals that perceived usability, PS and AS have a total effect
of 0.68, 0.11 and 0.29, respectively. For example, the total effect of perceived usability
was computed by summing the products of each path coefficient along the three
possible paths starting from perceived usability to reach intention to revisit, i.e.,
0.42 0.39 0.29 + 0.62 0.29 + 0.46. Similarly, the total effect of PS 0.11 was
obtained by multiplying the path coefficient 0.39, which linked PS to AS, by 0.29, which
linked AS to intention to revisit.
Perceived usability has the highest total effect 0.68 in the three constructs of
perceived usability, PS and AS and is thus the most important driver of intention to
revisit.
6 Conclusion
6.1 Explanation
For a successful government website, users should be satisfied when using it and be
willing to continuously revisit the website as needed. Measurement and evaluation
provides guidelines to improve e-government websites. In this paper, we measured
usability of e-government websites and found major factors affecting the usability using a
user testing method. In addition, we investigated the causal effects among the constructs
of perceived usability, user satisfaction and revisiting intention. The target websites were
the Korean e-government web portal and 18 websites linked by the web portal, which
represents Korean administration departments.
The overall usability score of Korean e-government websites was not high.
In particular, the site design score was low although contents design scored well.
We found six factors affecting usability for e-government websites, which were
navigation, utilisation of images and graphics, effective readability, utilisation of
multimedia technology, site structure and information search.
16 D-H. Byun and G. Finnie
The perceived usability strongly affected both user satisfaction and intention to
revisit. It also affected intention to revisit directly without the intermediation of user
satisfaction.
6.2 Implications
Although considerable research for e-government success has been performed, there has
been a lack of research on evaluation and improvement of e-government websites.
E-government websites including the portal site play a major role in the interaction
between government and citizens. Given that the primary goal of e-government is
efficient interaction with citizens via quality websites, relevant methods for evaluating
and measuring e-government websites are important.
The major contribution of this paper was first to adapt the usability concept for
evaluating e-government websites. Even though a reasonably well-established set of
usability factors are generally accepted for commercial website evaluation, we suggested
special factors relevant for e-government website evaluation. Therefore, our approach can
provide guidelines to improve e-government website design, as well as practically be
used as a method for ranking e-government websites. In addition, the user testing method
performed in this paper allows the production of exact evaluation results with relatively
less respondents, which is an advantage over other exploratory approaches.
Second, the concept of intention to revisit e-government websites has rarely been
considered as a factor for e-government website success. The finding of the relationship
between intention to revisit and usability is a major contribution of this paper. Compared
with the Singapore research performed by Teo et al. (20082009), we obtained the same
result in the Korean case that user satisfaction was a strong driver to induce revisiting
e-government websites. We classified the user satisfaction concept into AS and PS,
which differs from the research of Teo et al. (20082009). Our work implies that
perceived usability was the most important driver for affecting intention to revisit for
e-government users even though PS was a strong factor, which increased revisiting
intention. Some users who tried to achieve their goals on the website were affected by
lack of PS. These users did not have an intention to revisit if they were not physically
satisfied with the website, even though they successfully achieved their goal on the
websites.
In conclusion, our work proved that usability is an important criterion for
e-government website evaluation, as well as perceived usability leading to intention
to revisit. In addition, our work indicates that making website users satisfied physically
was more important than achieving their goals on the e-government websites. Because
e-government websites usually contain a larger volume of information than commercial
websites, users may find it hard to access useful information through navigation.
Sometimes, this makes users fatigued and users, therefore, will tend not to revisit
e-government websites.
6.3 Limitations
The limitations of this paper were first in the number of task items in the user testing.
Among the four types of questionnaire items suggested by Spool et al. (1999), we did not
use comparison of facts and questions asking comparison of judgements. Second, we did
not propose a complete method for ranking e-government websites using the significant
Evaluating usability, user satisfaction and intention 17
factors affecting usability. To do that, the determination of weight of the usability factors
is necessary and this will form the basis of further research. Third, we did not include
trust and service quality in the proposed model. To find further significant factors
affecting intention to revisit e-government websites, it will be necessary
to develop a structural model including usability, user satisfaction, trust and service
quality.
Acknowledgements
References
Anderson, J.C. and Gerbing, D.W. (1988) Structural equation modeling in practice: a review and
recommended two-step approach, Psychological Bulletin, Vol. 103, pp.411423.
Badre, A.N. (2002) Shaping Web Usability: Interaction Design in Context, Addison-Wesley,
Pearson Education, UK.
Baker, D.L. (2009) Advancing e-government performance in the United States through enhanced
usability benchmarks, Government Information Quarterly, Vol. 26, No. 1, pp.8288.
Bertot, J.C. and Jaeger, P.T. (2006) User-centered e-government: challenges and benefits for
government web sites, Government Information Quarterly, Vol. 23, No. 2, pp.163168.
Byrne, B.M. (1998) Structural Equation Modeling with LISREL, PRELIS, SIMPLIS, Lawrence
Erlbaum, Mahwah, Basic Concepts, Applications, and Programming, NJ.
Byun, D.H. (2005) Evaluation of the Korean e-government web sites focused on usability,
The Korean Journal of Information Systems Review, Korea, Vol. 7, No. 1, pp.120.
Byun, D.H. (2006) Usability factors and variables of e-government web sites, The Korean
Journal of Information Policy, Korea, Vol. 13, No. 3, pp.2748.
Byun, D.H. (2007) Perceived usability of e-government web sites affecting the user satisfaction
and revisiting, The Korean Journal of Information Systems, Vol. 16, No. 2, pp.5168.
Byun, D.H. and Jeon, H.D. (2006) Factor analysis of the usability for Korean e-government web
sites, The Korean Journal of Social Science Research Review, Korea, Vol. 22, No. 1,
pp.435456.
Casal, L., Flavin, C. and Guinalu, M. (2008) The role of perceived usability, reputation,
satisfaction, and consumer familiarity on the web site loyalty formation process, Computers
in Human Behavior, Vol. 24, No. 2, pp.325345.
Chen, S.Y. and Macredie, R.D. (2005) The assessment of usability of electronic shopping:
a heuristic evaluation, International Journal of Information Management, Vol. 25, No. 6,
pp.516532.
Choudrie, J., Wisal, J. and Ghinea, G. (2009) Evaluating the usability of developing countries
e-government sites: a user perspective, Electronic Government, Vol. 6, No. 3, pp.265281.
Delice, E.K. and Gngr, Z. (2009) The usability analysis with heuristic evaluation and analytic
hierarchy process, International Journal of Industrial Ergonomics, Vol. 39, No. 6,
pp.934939.
Deng, Z., Lu, Y., Wei, K.K. and Zhang, J. (2010) Understanding customer satisfaction and loyalty:
an empirical study of mobile instant messages in China, International Journal of Information
Management, Forthcoming issue, Vol. 30, No. 4, pp.289300.
Detlor, B., Hupfer, M.E. and Ruhi, U. (2010) Internal factors affecting the adoption and use of
government websites, Electronic Government, Vol. 7, No. 2, pp.120136.
18 D-H. Byun and G. Finnie
Fang, X. and Holsapple, C.W. (2007) An empirical study of web site navigation structures impact
on web site usability, Decision Support Systems, Vol. 43, No. 2, pp.476491.
Flavin, C., Guinalu, M. and Gurrea, R. (2006) The role played by perceived usability,
satisfaction and consumer trust on web site loyalty, Information & Management, Vol. 43,
No. 1, pp.114.
Floropoulos, J., Spathis, C., Halvatzis, D. and Tsipouridou, M. (2010) Measuring the success of
the Greek taxation information system, International Journal of Information Management,
Vol.30, No. 1, pp.4756.
Fornell, C. (1982) A Second Generation of Multivariate Analysis, Methods, Praeger Special
Studies, Vol. 1, New York.
Gotoh, R. (2009) Critical factors increasing user satisfaction with e-government services,
Electronic Government, Vol. 6, No. 3, pp.252264.
Henriksson, A., Yi, Y., Frost, B. and Middleton, M. (2007) Evaluation instrument for
e-government websites, Electronic Government, Vol. 4, No. 2, pp.204226.
Hernndez, B., Jimnez, J. and Martn, M.J. (2009) Key web site factors in e-business strategy,
International Journal of Information Management, Vol. 29, No. 5, pp.362371.
Hung, S.Y., Chang, C.M. and Yu, T.J. (2006) Determinants of user acceptance of the
e-government services: the case of online tax filling and payment systems, Government
Information Quarterly, Vol. 23, No. 1, pp.97122.
Lean, O.K., Zailani, S., Ramayah, T. and Fernando, Y. (2009) Factors influencing intention to use
e-government services among citizens in Malaysia, International Journal of Information and
Management, Vol. 29, No. 6, pp.458475.
MacCallum, R. (1986) Specification searches in covariance structure modeling, Psychological
Bulletin, Vol. 100, pp.107120.
Monique, W. and Jaspers, M. (2009) A comparison of usability methods for testing interactive
health technologies: methodological aspects and empirical evidence, International Journal of
Medical Informatics, Vol. 78, No. 5, pp.340353.
Nakamichi, N., Sakai, M., Shima, K., Hu, J. and Matsumoto, K. (2007) WebTracer: a new web
usability evaluation environment using gazing point information, Electronic Commerce
Research and Applications, Vol. 6, No. 1, pp.6373.
Nielsen, J. (1994) Heuristic evaluation, in Nielsen, J. and Mack, R.L. (Eds): Usability Inspection
Methods, John Wiley and Sons, New York, pp.2561.
Nielsen, J. (1996) Usability metrics: tracking interface improvement, IEEE Software, Vol. 13,
No. 6, pp.1214.
Nielsen, J. (2000) Designing Web Usability: The Practice of Simplicity, New Riders Publishing,
Pearson Education, UK.
Nunally, J. (1978) Psychometric Theory, McGraw-Hill, New York.
OCED (2003) The e-government imperative, OECD E-government Flagship Report,
GOV/PUMA(2003)6, 7 March.
Oztekin, A., Nikov, A. and Zaim, S. (2009) UWIS: an assessment methodology for usability
of web-based information systems, Journal of Systems and Software, Vol. 82, No. 2,
pp.20382050.
Palmer, J. (2002) Web site usability, design, and performance metrics, Information System
Research, Vol. 13, No. 2, pp.151167.
Sarmad, A. and Hamid, A. (2009) E-government evaluation: citizens perspective in developing
countries, Information Technology for Development, Vol. 15, No. 3, pp.193208.
Smith, A.G. (2001) Applying evaluation criteria to New Zealand government web sites,
International Journal of Information Management, Vol. 21, pp.137149.
Spool, J.M., Scanlon, T., Schroeder, W., Snyder, C. and DeAngelo, T. (1999) Web Site Usability,
A Designers Guide, Morgan Kaufman Publishers Inc., San Franciso, CA, USA.
Evaluating usability, user satisfaction and intention 19
Srivastava, S.C. and Teo, T.S.H. (2007) E-government payoffs: evidence from cross-country data,
Journal of Global Information Management, Vol. 15, No. 4, pp.2040.
Srivastava, S.C. and Teo, T.S.H. (2008) The relationship between e-government and national
competitiveness, the moderating influence of environmental factors, Communications of the
Association for Information Systems, Vol. 23, pp.7394.
Teo, T.S.H., Srivastava, S.C. and Jiang, L. (20082009) Trust and electronic government success:
an empirical study, Journal of Management Information Systems, Vol. 25, No. 3, pp.99131.
Verdegem, P. and Verleye, G. (2009) User-centered e-government in practice: a comprehensive
model for measuring user satisfaction, Government Information Quarterly, Vol. 26, No. 3,
pp.487497.
Wescott, C.G. (2001) E-government in Asia-Pacific region, Asian Journal of Political Science,
Vol. 9, No. 2, pp.124.
West, D.M. (2007) Global E-Government, 2007, Center for Public Policy, Brown University, 2007,
from http://www.InsidePolitics.org/egovtdata.html
Wu, H., Ozok, A.A., Gurses, A.P. and Wei, J. (2009) User aspects of electronic and mobile
government: results from a review of current research, Electronic Government, Vol. 6, No. 3,
pp.233251.
Zhang, Y.J. and Hsieh, C. (2010) Chinese citizens opinions on e-government benefits, issues and
critical success factors, Electronic Government, Vol. 7, No. 2, pp.137147.
Zimmerman, D.E. and Muraski, M.L. (1995) Usability testing-an evaluation technique,
The Elements of Information Gathering, A Guide for Technical Communications, Scientists
and Engineers, Oryx Press, Phoenix, Ariz.