bij cover (i).

qxd 07/02/2006 10:22 Page 1

ISBN 1-84544-908-8 ISSN 1463-5771

Volume 13 Number 1/2 2006

Benchmarking
An International Journal

Performance measures,
benchmarking and best practices
in the new economy
Guest Editors: Angappa Gunasekaran and
Goran D. Putnik

www.emeraldinsight.com
Benchmarking: ISSN 1463-5771

Volume 13
An International Journal Number 1/2
2006

Performance measures, benchmarking

and best practices in the new economy
Guest Editors
Angappa Gunasekaran and Goran D. Putnik

Access this journal online __________________________ 3

CONTENTS
Editorial advisory board ___________________________ 4

Guest editorial ____________________________________ 5

Achieving business excellence through synchronous

supply in the automotive sector
David Bennett and James O’Kane __________________________________ 12

A benchmarking methodology for metalcasting

industry
Laura M.M. Ribeiro and J.A. Sarsfield Cabral ________________________ 23

An investigation of supply chain performance

measurement in the Indian automotive sector
Mohammed Saad and Bhaskar Patel _______________________________ 36

Best practices of collaboration between university

and industrial SMEs
P. Peças and E. Henriques ________________________________________ 54

Access this journal electronically

The current and past volumes of this journal are available at:
www.emeraldinsight.com/1463-5771.htm
You can also search more than 100 additional Emerald journals in
Emerald Fulltext (www.emeraldinsight.com/ft) and Emerald
Management Xtra (www.emeraldinsight.com/emx)
See page following contents for full details of what your access includes.
 Global benchmarking for internet and e-commerce
CONTENTS applications
continued A.M. Ahmed, M. Zairi and S.A. Alwabel_____________________________ 68

Value decomposition of e-commerce performance

Dieter Fink ____________________________________________________ 81

Action research and networking benchmarking in

developing Nordic statistics on woman
entrepreneurship
Paula Kyrö ____________________________________________________ 93

An expert diagnosis system for the benchmarking of

SMEs’ performance
Josée St-Pierre and Sylvain Delisle __________________________________ 106

Performance measures in English small and medium

enterprises: survey results
Sérgio D. Sousa, Elaine M. Aspinwall and A. Guimarães Rodrigues_______ 120

Performance measurement of AMT: a cross-regional

study
Carlo A. Mora Monge, S. Subba Rao, Marvin E. Gonzalez and
Amrik S. Sohal _________________________________________________ 135

Benchmarking performance indicators for banks

Chien-Ta Ho and Yun-Shan Wu ___________________________________ 147

SWOT analysis for Air China performance and

its experience with quality
A.M. Ahmed, M. Zairi and K.S. Almarri ____________________________ 160

Benchmarking web site functions

Hugues Boisvert and Marie-Andrée Caron ___________________________ 174

Knowledge-based benchmarking of production

performance
B. Denkena, R. Apitz and C. Liedtke ________________________________ 190

B2B international internet marketing:

a benchmarking exercise
R. Eid, M. Trueman and A.M. Ahmed ______________________________ 200

Comparative evaluation of practices: lessons from

R&D organizations
N. Mrinalini and Pradosh Nath ____________________________________ 214
www.emeraldinsight.com/bij.htm

As a subscriber to this journal, you can benefit from instant, Additional complimentary services available
electronic access to this title via Emerald Fulltext or Emerald Your access includes a variety of features that add to the
Management Xtra. Your access includes a variety of features that functionality and value of your journal subscription:
increase the value of your journal subscription.

How to access this journal electronically E-mail alert services

To benefit from electronic access to this journal, please contact These services allow you to be kept up to date with the latest
support@emeraldinsight.com A set of login details will then be additions to the journal via e-mail, as soon as new material
provided to you. Should you wish to access via IP, please enters the database. Further information about the services
provide these details in your e-mail. Once registration is available can be found at www.emeraldinsight.com/alerts
completed, your institution will have instant access to all articles
through the journal’s Table of Contents page at Connections
www.emeraldinsight.com/1463-5771.htm More information
about the journal is also available at www.emeraldinsight.com/ An online meeting place for the research community where
bij.htm researchers present their own work and interests and seek other
researchers for future projects. Register yourself or search our
Our liberal institution-wide licence allows everyone within your database of researchers at www.emeraldinsight.com/
institution to access your journal electronically, making your connections
subscription more cost-effective. Our web site has been
designed to provide you with a comprehensive, simple system Emerald online training services
that needs only minimum administration. Access is available via
You can also access this journal online. Visit
IP authentication or username and password.
www.emeraldinsight.com/training and take an Emerald online
tour to help you get the most from your subscription.
Key features of Emerald electronic journals
Automatic permission to make up to 25 copies of individual Choice of access
articles Electronic access to this journal is available via a number of
This facility can be used for training purposes, course notes, channels. Our web site www.emeraldinsight.com is the
seminars etc. This only applies to articles of which Emerald owns recommended means of electronic access, as it provides fully
copyright. For further details visit www.emeraldinsight.com/ searchable and value added access to the complete content of
copyright the journal. However, you can also access and search the article
content of this journal through the following journal delivery
Online publishing and archiving services:
As well as current volumes of the journal, you can also gain EBSCOHost Electronic Journals Service
access to past volumes on the internet via Emerald Fulltext or ejournals.ebsco.com
Emerald Management Xtra. You can browse or search these Informatics J-Gate
databases for relevant articles. www.j-gate.informindia.co.in
Key readings Ingenta
This feature provides abstracts of related articles chosen by the www.ingenta.com
journal editor, selected to provide readers with current awareness Minerva Electronic Online Services
of interesting articles from other publications in the field. www.minerva.at
OCLC FirstSearch
Non-article content www.oclc.org/firstsearch
Material in our journals such as product information, industry SilverLinker
trends, company news, conferences, etc. is available online and www.ovid.com
can be accessed by users.
SwetsWise
Reference linking www.swetswise.com
Direct links from the journal article references to abstracts of the
most influential articles cited. Where possible, this link is to the
full text of the article.
Emerald Customer Support
For customer support and technical help contact:
E-mail an article E-mail support@emeraldinsight.com
Allows users to e-mail links to relevant and interesting articles to Web www.emeraldinsight.com/customercharter
another computer for later use, reference or printing purposes. Tel +44 (0) 1274 785278
Fax +44 (0) 1274 785204
Structured abstracts
Emerald structured abstracts provide consistent, clear and
informative summaries of the content of the articles, allowing
faster evaluation of papers.
BIJ EDITORIAL ADVISORY BOARD
13,1/2 David Bennett Gopal Kanji
Aston University, UK Sheffield Hallam University, UK
Walter W.C. Chung Archie Lockamy III
The Hong Kong Polytechnic University, Samford University, USA
Hong Kong Kambiz Maani
4 Sylvia Codling University of Auckland, New Zealand
The Benchmarking Centre Ltd and Professor Rodney McAdam
Oak Business Developers plc, UK University of Ulster, School of Business,
John F. Dalrymple Organisation and Management, Belfast, UK
RMIT Business School, Melbourne, Australia Christian Madu
Tom Dolan Pace University, USA
President, The Benchmarking Exchange, Jaideep Motwani
USA Chairman, Benchmarking Committee, Grand Valley State University, USA
ASQC, USA
Richard Schonberger
James R. Evans Schonberger & Associates, USA
University of Cincinnati, USA
Dean M. Schroeder
Barbara Flynn
Valparaiso University, USA
Wake Forest University, USA
Roger G. Schroeder
Shuichi Fukuda
University of Minnesota, USA
Director, Global Learning Center, Tokyo
Metropolitan Institute of Technology, Japan Joel D. Wisner
University of Nevada at Las Vegas, USA
H. Peter Holzer
University of Economics, Vienna, Austria Dr Yahaya Y. Yusuf
Zahir Irani Business School, University of Hull, UK
Brunel University, Middlesex, UK

Benchmarking: An International
Journal
Vol. 13 No. 1/2, 2006
p. 4
# Emerald Group Publishing Limited
1463-5771
 Guest editorial
Guest editorial
About the Guest Editors Angappa Gunasekaran is a Professor of Operations Management at
the Charlton College of Business, The University of Massachusetts – Dartmouth. He has widely
published in operations management and information systems. Currently, he is interested in
5
researching the areas of supply chain management, information technology/systems evaluation,
enterprise resource planning, benchmarking and logistics systems and management.
Goran D. Putnik is a Professor of Production Engineering at the Department of Production
Engineering, College of Engineering, The University of Minho – Guimaraes. He has widely
published in operations management and production engineering. Currently, he is interested in
researching the areas of supply chain management, information technology/systems evaluation,
agile manufacturing, new product development, enterprise resource planning, benchmarking and
logistics systems and management.

Performance measures, benchmarking and best practices in the new

economy
Performance measures, benchmarking and best practices are the critical success
factors towards achieving business excellence in the twenty-first century
organizational competitiveness. The major challenge, which companies face, is
developing suitable managerial tools for measuring performance, benchmarking and
best practices in the new economy. The new economy comprises of global market,
international competition, and physically distributed operations and selling goods on
the cyberspace. Reviewing the literature on the related topics, should be able to come to
a conclusion that there is a need for more research in the areas of performance
measures, benchmarking and best practices in the twenty-first century organizations.
Organisations employ performance measures and benchmarking to promote and
implement best practices in order to improve competitiveness. Considering the
importance of performance measures and benchmarking in organizational
competitiveness, this special issue attempts to cover the topics of performance
measures, benchmarking and best practices in new economy in all types of business
organizations. The objective of this special issue is to facilitate advances in the theory
and practice of performance measures and benchmarking and in turn to implement and
promote companies’ best practices in the new economy, at all organisational levels.

The Guest Editors would like to thank all the participants of the First International Conference
on Performance Measures, Benchmarking and Best Practices in New Economy (Business
Excellence 2003) which was held in Guimaraes, Portugal. Without the support of authors of the
selected papers, this special issue would not have been a reality. Also, the Guest Editors would
like to thank over 75 referees who reviewed manuscripts for this special issue. Special thanks go
to John Peters, Emerald Publishing Group Ltd who has been supportive of the Business
Excellence 2004 Conference and this special issue project. The Guest Editors of this special issue
would also like to thank Ms Rosie Knowles (Managing Editor, BIJ) for her constant
Benchmarking: An International
encouragement and excellent support throughout this special issue project. Finally, gratitude is Journal
expressed to the University of Minho (Portugal) and Dean Raymond Jackson of the Charlton Vol. 13 No. 1/2, 2006
pp. 5-11
College of Business, The University of Massachusetts – Dartmouth (USA) for their q Emerald Group Publishing Limited
encouragement and support to the conference and the special issue. 1463-5771
BIJ The term “new economy” has different interpretations. One of the most widely
13,1/2 accepted terms is that the new economy is the economy of “new technologies”, i.e. the
economy of electronic- and the internet-based services, e.g. e-commerce,
e-manufacturing, and e-business, together with the technology that supports them,
e.g. computers, telecommunication and supporting software. Today, it is difficult to
imagine best practices and business excellence without these tools. Therefore, we define
6 the term “new economy” as the economy of the future, performing best practices, where
performance measures and benchmarking are indispensable tools. It is the economy that
performs in previously unforeseen ways, whether it be the economy of “new
technologies” or the economy of “traditional technologies”, and where the best practices
and business excellence are all-important.
The papers that appear in this special issue are selected from a large pool of papers
presented at the First International Conference on Performance Measures, Benchmarking
and Best Practices in New Economy (Business Excellence 2003) which was held in
Guimaraes, Portugal from June 10-13, 2003. Over 125 delegates from more than 30
countries have attended the conferences and presented their papers. Furthermore, the
papers that appear in this special issue were subject to normal reviewing process of BIJ.
We hereby provide an overview of papers that appear in this special issue on performance
measures, benchmarking and best practices in new economy.
The paper, “Achieving business excellence through synchronous supply in the
automotive sector” by Bennett and O’Kane provides the reader with an appreciation of
the present trends in the UK automotive supply industry as they affect the first and
second tier supply chain and aims to give researchers an insight into the complex
interaction between suppliers and their customers involved. The UK automotive
industry is undergoing a fundamental change, with the main centre of attention on the
increasing integration of suppliers in the development of products and production
processes. Pre-assembled components for new car models are being outsourced to
external suppliers. While the supplier deals with the entire development process, from
the original prototype construction to process production, the vehicle manufacturer can
devote increasing attention to the core areas of business. This paper studies how
external influences, i.e. the end customer; legislation or natural changes cause, or force
the suppliers to re-assess their working needs, requirements and strategies. Also, the
implications of leading edge working practices that are necessary along the
downstream side of the supply chain to ensure minimum operating wastes and satisfy
best practice.
From the conceptual point of view, benchmarking seems a simple process. However,
in practice, a number of different models and approaches are used for assessing the
companies’ performance. Ribeiro and Sarsfield Cabral, in the first part of their paper,
“A benchmarking methodology for metalcasting industry”, some benchmarking
methodologies used for the performance assessment are analyzed. The planning stage
of a specific benchmarking methodology for the foundry industry is presented. This
methodology has been developed within the scope of the Analysis and Selection of
Benchmarking Methodologies for Metalcasting Industry project. This project was
supported by Portuguese Foundry Association, was structured in two stages:
(1) developing the model for the performance assessment (including performance
indicators specially designed for the foundry companies); and
(2) conducting a pilot benchmarking exercise in order to validate the model.
Both stages (1) and (2) covered eight relevant national foundries. The exercise endorses Guest editorial
the view that benchmarking is a time-consuming tool, demanding sound resilience and
certainly a strong commitment of the top-managers. The authors suggest that in order
to prevent a premature ending of the benchmarking exercise it is important to provide
a method for organizing the improvement practices in hierarchical manner.
The paper, “An investigation of supply chain performance in the Indian automotive
sector”, by Saad and Patel investigates the supply chain performance practices in a 7
developing nation through a case of Indian automobile sector. It discusses the motives,
enablers and inhibitors for the adoption of supply chain management concepts and
attempts to identify performance measures for supply chain performance
measurement. The relevant literature is reviewed and semi-structured interviews
and an exploratory survey are used as a basis for the analysis of supply chain
measurement in the Indian auto sector. The authors argue that there is an increasing
awareness about the need to collaborate with world class players and enhance
performance through the use of new management concepts. Their analysis suggests
that performance improvement be not seen as a common task based on shared learning
and joint problem solving.
Several major changes are currently taking place in the manufacturing industry all
around the world. These changes have introduced new competitive demands (both in
managerial and technological fields) stressing the adaptability requirements in the
relationships between business partners. To face the new demand, a mix of structured
knowledge and empirical experience will play an important role in building stronger
and more sustainable manufacturing systems. Universities must challenge the young
future engineers with authentic industrial experiences in order to build their practical
framework and encourage their entrepreneurial growth. On the other hand, to intensify
the innovation processes in small and medium enterprise (SME) companies a close
collaboration with the universities must be promoted in a way that fits both needs. The
paper, “Best practices of collaboration between university and industrial SMEs” by
Peças and Henriques presents the experience carried on by a university group fostering
the collaboration with SME companies involving young engineering students and
researchers in projects designed for the resolution of real industrial problems. They
proposed collaboration model and presented four real case studies. The results
obtained demonstrated several benefits and some of them include:
.
training young engineering students for an active problem-solving attitude,
within a systematic industrial perspective;
.
smoothness of the students transition to their professional life; and
.
promotion of a collaboration culture between SME and academia world for real
problems-solving and for continuous improvement and innovation processes.

The paper, “Global benchmarking for internet and e-commerce applications”, by

Ahmed, Zairi and Alwabel reports a global benchmarking exercise carried out to
examine issues related to the development of the internet and e-commerce in the
Kingdom of Saudi Arabia. It underpins the existing problems facing consumers and
organisations and the hidden opportunities, which will be exploited by overcoming
these obstacles. The significant of this research work is that consumers nowadays are
using the internet as part of their daily activities and this can range from just gathering
general information to purchasing products/services online. The authors have collected
BIJ data by survey and using standard questionnaires. The result of this survey indicates
13,1/2 that the more the awareness of using the internet the more benefit would be gained.
The survey also revealed some barriers related to internet usage from users, managers
and accountants perspectives. Although, the internet is no doubt useful, and timely, the
delivery of service in Saudi Arabia is costly and inefficient.
The technique of decomposition appears intuitively well suited to solving the
8 problem of developing performance measures for e-commerce. To produce a meaningful
measure of performance necessitates e-commerce values are decomposed into lower
levels until a “succinct statement” can be made about the value that e-commerce
produces. The paper, “Value decomposition of e-commerce performance”, by Fink
identifies value constructs and value variables for six e-commerce applications, which
are determined by their “internal/external to the business” nature and focus on
“customer/business”. It also develops values for corporate performance management
and incorporates them into an integrated value framework. The paper concludes that
although there is large agreement on the key values obtained from e-commerce, it is
difficult to achieve the degree of integrity required by the decomposition technique.
There are two key challenges to be met, namely achieving value splicing and functional
integrity.
Kyrö in her paper, “Action research and networking benchmarking in developing
nordic statistics on woman entrepreneurship”, develops a theoretical basis for the
concept and process of benchmarking. Considering the concept, the paper suggests that
the most recent developments in the content, forms and targets of benchmarking
actually provide us with a new form of benchmarking, namely networking
benchmarking. In networking benchmarking the participants create new best
practices, rather than adopt them from others. Regarding the process of
benchmarking, the study suggests that an action research approach provides an
alternative for conducting networking processes. Both of these, the new networking
benchmarking concept and the action research approach, extend the current theoretical
basis for benchmarking, thus opening up potential directions for future scientific as well
as practical developments. In order to explore the possibilities of these new theoretical
constructions, this study applies them in a Nordic project, in which the representatives of
five Nordic countries studied together how to develop their national statistical
procedures on woman entrepreneurship by benchmarking and developing best
practices for use in the Nordic countries. The results indicate that networking
benchmarking and the action research model indeed have potential both as a theoretical
frame and a practical tool, thus encouraging their further development.
Although benchmarking has proven to be valuable for large businesses and
organizations for quite sometime, up until recently, serious doubts existed as to its
usefulness for smaller businesses. In the paper, “An expert diagnosis system for the
benchmarking of SMEs performance”, St-Pierre and Delisle present the PDG, a
benchmarking system meant to evaluate SMEs from an external perspective in order to
produce a diagnosis of their performance and potential, complemented with relevant
recommendations. Their research results show that benchmarking allows SMEs to
improve their operational performance thus confirming the usefulness of
benchmarking. These results also confirm the value of the recommendations
included in the PDG report concerning short-term actions to be undertaken to modify
management practices. They believe the PDG system proves that if the benchmarking
approach is tailored to SMEs’ needs, an adequate tool can be devised and used to help Guest editorial
SMEs maintain or increase their competitiveness.
The paper, “Performance measures in English small and medium enterprises:
survey results”, by Sousa, Aspinwall and Rodrigues presents a literature review and
discusses the methodology used. The objective of this work was to determine the
current state of knowledge related to performance measures and their degree of
implementation in SMEs in England. General opinions regarding performance 9
measurement in English SMEs are described, including the most important measures
and the biggest obstacles to the adoption of new ones. Hypotheses about differences
between groups are tested and discussed. This work concludes that there is a gap
between the theory/knowledge of performance measures and the practice in English
SMEs and that innovation and learning measures should be applied more widely.
Training of employees and difficulty defining new performance measures were
highlighted as the most important obstacles to the adoption of new performance
measures. Only a limited SMEs were applying statistical process control and cultural
change programmes.
Mora Monge, Subba Rao and Gonzalaez and Sohal in their paper, “Performance
measurement of AMT: a cross-regional study” examine the relationship of performance
in advanced manufacturing technologies (AMT) to the levels of AMT investments and
planning and implementation activities in AMT. The study was conducted in two
regions of America: Anglo-Saxon (U.S.A. and Canada) and Hispanic (Mexico and Costa
Rica). Drawing from an extensive survey of the literature on manufacturing strategy and
AMT, several factors emerged as possible predictors for performance. The question
ought to be answered in this research is: do AMT investments and planning and
implementation activities have an impact on Performance? And if so, are there
differences between the Anglo-Saxon and Hispanic regions? These questions are
answered using data from 222 firms, 125 Hispanic and 97 Anglo-Saxon.
In the paper, “Benchmarking performance indicators for banks”, Ho and Wu
construct a performance evaluation for banks with the stock performance taken into
consideration. If all financial indicators in the set are placed into the evaluation process,
data collection would be more difficult and resources would be wasted. Grey relation
analysis (GRA), a new method for commerce is used to evaluate the relative
performance of three major banks in Australia. The purpose of using GRA is to reduce
the number of indicators by selecting representative indicators from financial
statement analysis. In addition, this paper compares the GRA results with the financial
statement analysis and shows that the same result can be obtained.
China geographically is similar with USA, the overall turnover represent 7 percent
of that of USA; and in term of passengers uplifted in 2001 China airlines was similar
with UK’s, but the structure was totally different. China Airlines operations are mainly
confined within domestic market, more than 90 percent traffic from domestic market,
contrasting with British airlines 25 percent. The company is operating 339 routes,
which consists of 53 international and 286 domestic, operating more than 1,000
scheduled flights on weekly basis, serving 29 cities in 19 countries. About 66 percent of
its revenue was from the domestic market. Over the last two years, Air China has
experienced dramatic changes in both international and domestic market. Total
quality management (TQM) model in the early 1980s has rejuvenated it and become a
recognised approach necessary to achieve greater efficiencies, improved reputation
BIJ and continuing growth. If TQM is to be implemented within Air China, an
13,1/2 understanding of the long-term impact on the company’s future and commitment by
senior management to ensure effective leadership and teamwork are all needed
ingredients to success. The paper, “SWOT analysis for Air-China performance and its
experience with quality”, by Ahmed, Zairi and Almarri reports on the learnt lessons
during the implementation of TQM principles in Air China operations. The objective is
10 to realize the quality-strategy integration that needs to be seen throughout the process
of strategy management.
Benchmarking web site functions involves measuring and comparing functions’
development. The paper, “Benchmarking web site functions”, by Boisvert and Caron
presents the first phase of a more comprehensive research project aimed at evaluating
and documenting the impact of the internet on business processes. This paper presents
the rationale for web site function benchmarking, along with the methodology, and
main results of this exercise. They also suggest how companies could plan web site
functions development. The methodology developed for this benchmarking exercise
includes the development of an approach to describe and measure web site functions,
data collection logistics and a data analysis model. The main results provide a method
to identify directions of web site development, a typology of web sites by development
profile, and benchmarks and comparisons of web site function development.
Since benchmarking has become a common approach to optimize production
processes by comparing certain aspects of a company with its competitors. However,
one of the biggest challenges is not only to define suitable benchmarking topics and
partners, to gather and statistically evaluate characteristic data, but also to derive
concrete measures to interpret the results, i.e. to overcome the revealed weaknesses.
The paper, “Knowledge-based benchmarking of production performance”, by
Denkena, Apitz and Liedtke presents a successfully implemented and used
functional benchmarking methodology for production performance of small and
medium batch size processes, that is currently extended by using a knowledge base for
reasoning strategies to semi-automatically support the interpretation of the extracted
statistical data. The knowledge-based concept allows using sophisticated
interpretation strategies of an existing base of real company data. The decisive
point of the approach presented is to map production variables on room for
improvement by taking varying parameters into account. The proposed tool is a
valuable tool that takes advantage of a statistically firmed analysis of a substantial
database and combines it with the comprehensive expertise of experienced specialists
in the field of performance assessment.
B2B internet marketing has emerged as one of the key drivers of change and
successful business practice. However, such developments affect the dynamics of
traditional marketing as revealed in this recent survey of 123 UK industrial companies.
The study, “B2B international internet marketing: a benchmarking exercise”, by Eid,
Trueman and Ahmed provides the foundation for further research and good practice in
benchmarking since it combines the best aspects of traditional and new internet
marketing activities. Here 33 critical success factors were found to influence internet
marketing practice; from strategy and web site design to global dimensions and
corporate business environment. This study provides theoretical foundation for
research into global B2B internet marketing, as well as a guide for practitioner
decision-making if they are to remain competitive in the dynamic electronic Guest editorial
marketplace.
The paper, “Comparative evaluation of practices: lessons from R&D organizations”
by Mrinalini and Nath focuses on an international benchmarking study of R&D
organizations. This paper highlights some critical issues related to indicators that are
used to compare organizational practices. It is argued that for evaluation of
organizational practices, in many cases, depending upon quantitative indicators is 11
neither possible nor even desirable. In case of quantitative indicators much of the
analytical rigour of the benchmarking study could be lost. It is argued that if at all any
need for quantitative indicators arise it is at a much later stage of the benchmarking
exercise. Qualitative understanding of the problem is the most desirable aspect of the
organizational benchmarking exercise, which is complimented and supported with the
quantitative performance indicators.
Angappa Gunasekaran and Goran D. Putnik
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

BIJ
13,1/2 Achieving business excellence
through synchronous supply in
the automotive sector
12
David Bennett and James O’Kane
Newcastle Business School, University of Northumbria,
Newcastle upon Tyne, UK

Abstract
Purpose – To provide the reader with an appreciation of the present trends in the UK automotive
supply industry as they affect the first and second tier supply chain and aims to give researchers an
insight into the complex interaction between suppliers and their customers.
Design/methodology/approach – Literature review which is divided into the sub-sections of
strategy, structure, processes and co-ordination required within the manufacturing environment, to
identify the influence of synchronous supply as it affects these main areas of operations management.
Findings – Through the use of examples from previous papers written on automotive and related
supply industry operating methods, and to date Grounded Theory forming a pilot study to empirical
research for this research, it concludes that synchronous supply can provide the lead for an efficient
automotive supply chain.
Research limitations/implications – The concept of synchronous supply presents a challenging
research framework for supplier operating strategy, structuring processes and co-ordination between
the vehicle manufacturer (VM) and the whole supply chain.
Practical implications – A useful source of information for VMs and their first tier suppliers to
understand more about synchronous supply and its affect on the supply chain.
Originality/value – This paper presents the concept of synchronous supply as a challenging
framework for supplier operating strategy, structuring processes and co-ordination between the VM
and the whole supply chain.
Keywords Supply chain management, Automotive industry, Business excellence
Paper type Conceptual paper

Benchmarking: An International
Journal
Vol. 13 No. 1/2, 2006
pp. 12-22
q Emerald Group Publishing Limited
1463-5771
DOI 10.1108/14635770610644547
Introduction
With rapidly changing and under very competitive circumstances, the design,
development and marketing of new products with creative and innovative features are
essential for any company’s survival. For new product development to be effective, a
systematic approach to the understanding of customer requirements is necessary. These
requirements should then be firmly attached to future supplier’s product processes.
Within the automotive industry the matching and synchronisation of future
complex sub-assemblies and full systems needs to be managed carefully between
suppliers and supplier-customer to ensure project success. The innovative product
development and production process needs an understanding of continuously
changing customer wants and needs.
The literature review of this paper is divided into the sub-sections of strategy,
structure, processes and co-ordination (SSPC) required within the manufacturing
environment, to identify the influence of synchronous supply as it affects these main
areas of operations management. The aim has been to identify from available literature
variation between SSPC and the future working needs and trends in liaisons between Achieving
automotive suppliers and their customers. The way that other industries have changed business
dramatically over the past few decades may give the indication as to the future trends of
synchronisation in the UK automotive supply industry and if they affect any specific excellence
area of SSPC. However, the modern market place is highly varied and cannot be serviced
effectively by a single supply chain paradigm (Childerhouse and Towill, 2000).
Strategies have moved some suppliers in the automotive industry into a different 13
field of activity; such as the tier 0.5 supplier, a phrase originally attributed to Magna
International. The assumption being that the vehicle manufacturers (VMs) have
empowered the large first tier suppliers to completely develop and produce modules
(large sub sections) for their vehicles. This has led to suppliers seeking collaborative
relationships or supplier alliances or mergers.
Within the structure, or the inter-linking supporting framework to ensure that the
whole business relationship is compatible, different social, cultural and economic
contexts identify variations between suppliers. These variations can have a marked
effect on how the supplier performs in business with its customers.
Processes stated within the literature review, refer to the method of operation in
manufacture within the UK automotive supply industry and identifies research that
shows that manufacturing decision making can start earlier in the process. As leading
first and second tier suppliers move closer towards supplying full systems and
modules to VMs, the controlling processes to achieve total system compatibility must
become ever more critical to ensure project success.
The co-ordination of projects is embracing a form of partnership of which the
precise definitions may differ between suppliers and customers. There is evidence to
show that the VMs are pursuing a more intensive and interactive relationship with
their suppliers, collaborating in areas such as product development, supplier
development and information sharing.
This paper seeks in part, to provide the reader with an appreciation of the present
trends in the UK automotive supply industry as they affect the first and second tier
supply chain and inter link with synchronous supply. It aims to give the reader an
insight into the complex interaction between suppliers and their customers, who are
involved in such necessary relationships to ensure success in today’s competitive
automotive environment.

Synchronous supply – strategy related continuous improvements

Many theoretical works have been published emphasising the importance of a strategic
management of the manufacturing function and the management of quality in order to
gain competitive advantage. However, the competitiveness of a company is mostly
dependent on its ability to perform well in dimensions such as cost, quality, delivery,
dependability and speed, innovation and flexibility to adapt itself to variations in
demand (Carpinetti et al., 2000). While alignment of the manufacturing function with
strategic priorities is core to competitiveness, the continuous improvement of the
manufacturing function plays a very important complimentary role in the quest of
competitiveness in the long run. Continuous improvement has been defined as a
company wide process of focused and continuous incremental innovation (Bessant
et al., 1994). It is thus complimentary to the more radical change normally considered
under the term re-engineering.
BIJ However, many companies, in their attempts to rapidly adopt world-class
13,1/2 management practices such as TQM and many other acronyms, tend to devote little
or no attention at all to the impact of such practices on company strategic objectives,
market demands or even performance against competition. This misalignment between
operational management practices and competitive strategy can be listed as one of the
reasons for unsuccessful TQM implementations (Tatikonda and Tatikonda, 1996).
14 Synchronous supply is essentially a process where components supplied are
matched exactly to the production requirements of the customer (Doran, 2001).
However, to ensure that the best performance is attainable, the implementation of
Agile manufacturing into the operation method would give an added emphasis for
success.
Agility means being able to reconfigure operations, processes and business
relationships efficiently, while at the same time flourishing in an environment of
continuous change. Companies, and this includes automotive suppliers, need to open
their collective minds to a paradigm shift in how they design, manufacture and market
their products. Co-operation amongst suppliers must improve to support the need for
quick decision making and these suppliers must work together to achieve the overall
goal of improving manufacturing.
This would ensure a gateway for the introduction of synchronous supply but, in
addition to being agile the supplier also needs to maintain a leanness of its operation.
This requires the development of a value stream to eliminate all waste, including time
to enable a level schedule to be produced.
There have been three major phases or paradigm shifts of industrial production in
the modern world (Womack and Jones, 1990) and each phase was brought in by an area
of the world, which mastered its implementation. Craft production was mastered and
dominated by Europe. Mass production was mastered and dominated by the USA and
lean/JIT production has been mastered and dominated by the Japanese. Agile
production, the next phase in manufacturing, would appear to be the next requirement
for world-class manufacturing performance and a necessary requirement for
synchronous supply. For the future, it would seem that any industrial nation that
does not anticipate and enclose the next paradigm shift in manufacturing, risk’s losing
ground in manufacturing dominance.
A system that can move quickly amongst product models or between product lines
in a real-time response to customer demand will pave the way for synchronous supply.
To build agile manufacturing into the strategy of an enterprise is to ensure that it can
compete effectively with competitors against order requirements. Hooper et al. (2001)
go on to suggest that the future development of agile manufacturing will require
academics to evaluate modern manufacturing management solutions and consider
their adoption for agility.
To survive in today’s severe global market competition, enterprises need to
constantly improve quality and reduce cost. The investment in product research and
development is viewed as a strategic weapon capable of accomplishing this objective
and thus ensuring market share (Wei et al., 2000).

Synchronous supply strategy – customer requirements and satisfaction

QS 9000 Quality Standard provides a quality management system that focuses on
continuous improvement, defect prevention, reduction of variation and reduction of
waste in the supply chain thus increasing customer satisfaction and reducing quality Achieving
cost. In today’s automotive industry a first tier supplier will struggle to win business business
unless their system is certified to QS 9000 (Bramorski et al., 2000).
This situation has now cascaded itself down to the second tier supplier, in as excellence
much as the major first tier suppliers would expect the second tier to have fully
approved quality standards in place. Failure of the second tier supplier to agree to
do this could mean de-selection. As the automobile becomes ever more complex 15
with the accepted standard of JIT, the necessity of total quality is a requirement to
ensure minimum defects. It also paves the way for full confidence in synchronous
supply and module supply (larger assemblies of associated assemblies supplied as
one unit).
As an example, whilst supplying to Nissan Motor Manufacturing (UK) (NMUK) is a
demanding exercise for first tier suppliers that are required to supply on a JIT basis,
the degree of difficulty is compounded within the context of synchronous supply.
Doran (2002) suggests that whilst case studies and definitions abound extolling the
virtues and operations of JIT, there is little that explores what is perhaps the most
efficient and difficult type of supply.
If it is now accepted that critical to the success of a synchronous supply system is a
zero reject quantity and a close co-ordination of order scheduling, such a necessary
complex production environment as synchronous supply requires superior planning
and control to ensure that the customer requirements are met first time and every time.
Failure to ensure this could have the severe consequences of stopping the VMs
production line. A totally unacceptable situation, especially as stoppage costs would be
re-charged back to the supplier.
The need for productivity in the automotive industry and within its suppliers is
due to the high level of worldwide competition in this sector. Small and medium
enterprises (SMEs) play an important role in modern economies, especially the
second and third tier automotive supplier, because of their flexibility and ability to
innovate.
However, a large number of SMEs operate with poor forecasting and planning
systems and operate with long cycle times. They also can have problems with
unreliable inventory control systems, with no stock tracing and poor cost control. This
can lead to excess obsolete stock and eroding customer service levels (Gunasekaren
et al., 2000).
Electronic ordering appears to be the trend for the future. The use of
inter-organisational linkages such as electronic data interchange (EDI) has enabled
and reduced the co-ordination costs of increasingly market driven organisations and
electronic linkages such as EDI are becoming the necessary condition of doing
business with larger companies (Attaran, 2001).
NMUK utilise electronic linkages with their synchronous suppliers. The time from
the vehicle chassis passing the sensor point to the delivery on the production line is
about 2 hours 30 minutes, requiring NMUKs circle of synchronous suppliers to be
situated on the same business park. To avoid any fear of non-delivery NMUK insist
that these suppliers set up very local to the manufacturing plant within a radius of
3-4 miles.
This requires combining flexibility with predictability with a view to achieving
mass customisation (Kumar and Midha, 2001).
BIJ Synchronous supply structure – lean/agile production and logistics
13,1/2 With the advent of Japanese automotive manufacturers setting up their European
businesses in the UK, starting with Nissan in 1989, this has completely changed
traditional working practices in the UK. Following Nissan Motor Manufacturing at
Sunderland, Honda UK at Swindon and Toyota at Derby, all introducing Japanese
working practices at these manufacturing sites and cascading their operating
16 principals to the first tier suppliers who have further cascaded them through the
supply chain. The traditional UK automotive manufacturers such as MG Rover, Land
Rover and Jaguar were quick to adopt the Japanese working practices within their own
organisations for fear of falling behind state of the art procedures and placing
themselves in an uncompetitive situation.
Nissan at Sunderland are now the most efficient vehicle manufacturing plant in
Europe and are a world class manufacturer comparable with their parent plant in
Japan. Their success has also attracted a number of world class suppliers from Japan
and the US who now have satellite plants in the UK further strengthening the
automotive supply industry in the UK but also forcing the indigenous UK automotive
supply industry to quickly adapt new systems and procedures for fear of losing
traditional business.
If only one supplier has insufficient capacity to make it impossible to build a
finished vehicle at the VM, it is the one supplier with the least flexible resources that
will determine the overall scope for any capacity increase or volume flexibility
(Holweg, 2002). With this in mind, to achieve a synchronous supply do all the suppliers
within the supply chain need to supply synchronously for best effect both functionally
and economically?
This paper, sets out to determine the cut off point, or points, to which the supply
chain supplies synchronously yet identifying the different levels of process
configuration that are economically necessary for all suppliers to work to without
undue pressure or “hidden” process requirements placed upon them.
Only in achieving supply chain integration using the most appropriate production
methods will the optimum lead time be achieved through the supply chain and all
material and product flow synchronised to maximum effect. By comparison, the 1980s
traditional uncoordinated supply chain had long lead times and functional silos
resulting in multiple decision points, a lack of information transparency and minimal
synchronisation (Childerhouse and Towill, 2000).
Timing is becoming ever more critical, such that at the Nissan Spain (NMISA) Plant,
modules (larger assemblies of sub-assemblies of the vehicle) have a turn-around time of
supplier receipt of order to delivery of the module onto the vehicle of 90 minutes. This
very short order time must be maintained due to the physical size of the module
creating a zero storage situation either at the supplier or customer.
Perhaps the main difference between JIT and synchronous supply is the issue of
supplying the exact requirements of the customer reflecting all the various possible
number of options that may be required.
At a first tier synchronous supplier to NMUK, the EDI schedule from the VM to
the supplier allows for a pre-build of common sub-assembly components. This
would tend to follow a JIT supply pattern. The synchronous part of the production
schedule gives a 2-hour window from the receipt of electronic order to delivery into
Nissan.
 There seems to have been very little empirical research into the impact of e-business Achieving
on the supply chain configuration and that the use of electronic commerce for business
maintenance, repair and operating items will offer potential cost savings in the process
search, order and payment. Thus the internal procurement process would become excellence
leaner (Croom, 2001). However, managing a synchronous supply system is
operationally and logistically difficult, but, the system can ensure a number of
benefits for both the buyer and the supplier, which can centre around operational 17
efficiencies and cost.
A disciplined approach must also be maintained with other suppliers further down
the supply chain to ensure that a smooth and consistent supply of sub-components are
available for assembly. Second and third tier suppliers may not necessarily need to be
keyed up for synchronous supply, they may be de-coupled within the supply chain and
their products possibly supplied into a distribution/supplier warehouse to be delivered
into the first tier synchronous supplier. This could even achieve the lower cost if the
lower tier supplier did not need to have the facility to synchronise. Indeed as
Childerhouse and Towill (2000) advise, present day market places are very varied and
cannot be effectively supplied by a single supply chain paradigm.
Although NMUKs system is at the forefront of automotive synchronous supply
systems, with Nissan only a very small core of high value units are supplied in this
manner. Many more are supplied on a JIT basis. A further point to note here, is that
NMUK only interface with their first tier suppliers electronically and directly. It is the
first tier supplier’s responsibility to ensure the availability of production parts from
second and third tier suppliers. Research by this author also reveals that this seems to
be a common practice in the UK automotive industry.
An empirical study of the North American automotive supply industry, found that
several companies surveyed (from 57) relied heavily on concurrent engineering to
achieve significant reductions in new product development times. However, few
empirical studies have examined the relationship between time-based performance and
overall business performance. Supplier partnering is advised as the single most
important factor explaining an increase in the speed of new product development.
Nevertheless, the empirical evidence linking continuous improvement to cycle time
reduction is very limited (Jayaram et al., 2000).

The evolution of production models: is a new paradigm emerging?

Bartezzahi (1999) asks this question and suggests that lean production is a myth,
rather than a faithful representation of the real processes of change. Leanness as a
concept itself is questioned. Its undesirable effects include the lack of young labour
willing to work in factories, the excessive product variety, the extreme pressure on
suppliers, the inability to find funds for new product development and the increase in
traffic congestion.
A case study of Japanese multinational companies in the UK identifies three
ingredients to the Japanese system of management, lean production system, total
quality management and human resource management. These are inter linked in order
to produce a total effect on the management of Japanese enterprises. However,
sequential just-in-time is not always needed. When the products are standardised and
relatively inexpensive it may not be too important to achieve the highest level of JIT
(Basu and Mirchnik, 1999). To highlight this point Figure 1 shows the “full circle
BIJ 3rd Tier raw material supply
3rd Tier RAW
13,1/2 MATERIAL
= MASS PRODUCTION
MASS 2nd Tier 2nd Tier supply
PRODUCTION JIT/LEAN
JIT/LEAN
1st Tier
Agile
18 0.5 Tier MODULE
SEQ.SYNCHRO
VM. (Man’f Operation)
VM SEQ. SYNCHRO
DISTR’
MASS
STOCK VM DISTRIBUTOR
MASS STOCK HANDLING
(FINISHED VEHICLE FOR
DESPATCH)

Figure 1. 1st Tier

Full circle diagram of AGILE
manufacturing VM OPERATION
0.5 Tier
SEQUENCE SYNCHRO
technologies used in OPERATION MODULE
vehicle manufacture SEQ/SYNCHRO

diagram” of manufacturing technologies used in vehicle manufacture. This writer

suggests that to achieve synchronous supply within the VMs operation other
technologies may be equally important to support the present progress of the supply
chain. As the lead time from consumer order to vehicle delivery reduces through
business necessity then synchronous supply may spread to other areas of the supply
chain.
Fractal manufacturing partnerships (FMPs) is an emerging approach designed to
maximise the logistical attributes of a lean production system. Noori and Lee (2000),
advise that VW pioneered a revolutionary approach aimed at maximising the
benefits of partnerships and proximity aimed at maximising the integration of
suppliers as assemblers within the VMs manufacturing facility. This elevates the
philosophy of “assembly within assembly”. In this case the supplier rents space in
the VMs plant and assembles the products directly to the vehicle. FMP is designed to
combine the logistical attributes of lean production with the strategic configuration
of agile network capabilities. However, this is not a general practice; the industry is
far from an open book trusting relationship required for integrated partnerships to
succeed.
In improving co-development through process alignment Evans and Jukes (2000)
found that there had been a recent surge of enthusiasm within the automotive industry
to build closer supplier relationships within the area of product development, however,
research into the implementation of co-development remains limited. A knowledge gap
existed between the co-developing partners, with suppliers suggesting that
communication was typically one-way. It is now recognised that suppliers have
become an essential and significant part of product development within today’s
automotive industry, with purchased parts from suppliers accounting for over 70
per cent of the entire vehicle production cost.
Supply chain collaboration and co-ordination Achieving
In a paper entitled “Sustaining strategic supplier alliances” Stuart and McCutcheon business
(1996) noted that the trend to outsource more materials and services had made firms
more reliant on their sources of supply. This has led to a move towards increasingly excellence
collaborative supplier relationships, referred to as supplier alliances. Widespread use
of these relationships is new and relatively little is known about the attribute that may
promote success or failure. The data that they collected shows that there is partial 19
support for the idea that alliance relationships will tend to produce better results with
time. With increased interdependence over time, information exchange becomes a key
element in alliance success (or failure).
The achievement of a sustainable competitive advantage has long been the goal of
companies and organisations. Over 80 per cent of Toyota’s achievement of competitive
advantage over its UK based rivals is based on the effective use of its supplier network.
When Toyota’s Japanese supply chain competitive advantage is analysed, it shows
that 18 per cent is directly derived from their internal competitive advantage with over
40 per cent coming from the first tier suppliers and a further 42 per cent coming from
the lower tier component suppliers.
The key to Toyota’s success would appear to be their highly effective supplier
integration process that over the past 50 years has enabled the excellence of their
internal hoshin kanri (policy deployment) strategic total Toyota Production System to
be shared with their direct suppliers. This is also outsourced to second and lower tier
suppliers. The primary method by which this is achieved is the kyoryoku kai (supplier
association), though the spread of knowledge and expertise about how to operate
supplier associations in the West has until recently been very limited (Hines and
Rich, 1998).

Co-ordination – for competitive advantage

Although supplier associations are fairly new within the western setting they are
providing significant benefits to the firms involved both as customers and suppliers.
Hines and Rich (1998) believe that the approach will continue to develop and evolve
and will help provide an effective framework for the development of world class supply
chains.
VMVW has pioneered the approach (FMP) involving the integration of selected
suppliers directly in their manufacturing plant in the Czech Republic. Thus the
supplier performs design, manufacture and assembly operations in close proximity to
the VM assembly line. However, research suggests that the transition to co-operative
relationships between buyers and suppliers may be more difficult for automotive
companies due to the high levels of complexity, compatibility, long lead time and
ingrained adversarial supplier relationships of the past (Ratnasingham, 2001). In
support of this Arkader (2001) found that in the Brazilian Automotive Industry,
suppliers identified in Buyers’ organisations a serious lack of communication between
purchasing, engineering and manufacturing, which acted against integrative practices
required in lean production systems. Brazilian suppliers considered that their historical
relationships with buyers were cordial, a situation perceived to have changed with the
evolution of supply relations during the 1990s.
The long-term customer retention in competitive markets requires the supplier to go
beyond the mere basic satisfaction and to look for ways of establishing ties of loyalty
BIJ that will help ward off competitor attack. It makes sense to look at customer loyalty
13,1/2 directly, simply because it reflects what happens in real life. Loyalty is retention with
attitude (Clarke, 2001).

Conclusions
This paper presents the concept of synchronous supply within the UK automotive
20 supply industry as a relatively new and emerging form of collaboration between VM
and their selected suppliers. The concept of synchronous supply presents a challenging
framework for supplier operating strategy, structuring processes and co-ordination
between the VM and the whole supply chain.
Through the use of examples from previous papers written on automotive and
related supply industry operating methods, and to date grounded theory forming a
pilot study to empirical research for this research, it would seem reasonable to conclude
that synchronous supply can provide the lead for an efficient automotive supply chain.
This in turn can provide the role for lean and agile production. In order to be successful
synchronous supply needs to encompass the latest lean and agile processes to ensure
that the significant amount of integration and trust required for success is maintained
between the VM and their associated suppliers.
Synchronous supply can provide the ideal opportunity for VMs and their first
tier suppliers to improve their product and process design and operational
communication and joint research and development. Ventures between VMs and
suppliers are becoming more common. However, if VMs want to enlist suppliers to
design for manufacture and assembly, then perhaps they should require suppliers
to assemble the product directly onto the vehicle to gain maximum efficiency of
process. Integration of supplier delivery and product design are two components of a
fractal configuration and many Japanese VMs and suppliers have achieved
communication and trust through the formation of affiliations with their suppliers
or kankei kaisha.
As VMs continue to standardise modular assemblies built around common vehicle
platforms, design for manufacture and flexible manufacturing systems will become
increasingly important. The ideal platform process would seem to require that the
design of the product and the design of the manufacturing process occur at the same
time, i.e. in synchronisation.
It must be remembered that it took Toyota decades to develop and implement
these practices that western suppliers are trying to introduce in a few years. This
is posing difficult changes to realise and throughout the supply chain. Figure 1
shows a “full circle diagram” of manufacturing technologies used in vehicle
manufacture. The argument here is that for synchronous supply to be a success
within VMs, the technique may not necessarily be applied across the full supply
chain.
Synchronous supply might be a panacea for selected first tier suppliers large
enough to support the VM with complicated modules and take on the 0.5 tier role. For
the majority of other suppliers in supporting first and second tier roles they are still
coming to terms with lean and agile processes. Third tier suppliers are often still in a
mass production scenario. Also the continued practice of mass stock holding of
finished vehicles before distribution to dealerships concludes the full circle of vehicle
manufacture through the supply chain.
 Further empirical research needs to be undertaken with the suppliers to determine if Achieving
the progress of synchronous supply will benefit a wider selection of suppliers to VMs business
and their selected module suppliers.
excellence
References
Arkader, R. (2001), “The perspective of suppliers on lean supply in a developing country
context”, Integrated Manufacturing Systems, Vol. 12 No. 2, pp. 87-94.
21
Attaran, M. (2001), “The coming age of online procurement”, Industrial Management & Data
Systems, Vol. 101 No. 4, pp. 177-81.
Bartezzahi, E. (1999), “The evolution of production models: is a new paradigm emerging?”,
International Journal of Operations & Production Management, Vol. 19 No. 2,
pp. 229-50.
Basu, D.R. and Mirchnik, V. (1999), “Strategic human resource management of Japanese
multinationals – a case study of Japanese multinational companies in the UK”, The Journal
of Management Development, Vol. 18 No. 9, pp. 714-32.
Bessant, J. and Caffyn, S. (1994), “Rediscovering continuous improvement”, Technovation, Vol. 14
No. 1, pp. 17-29.
Bramorski, T., Madan, S.M. and Motwani, J. (2000), “QS 9000 registration for lean
manufacturing”, The TQM Magazine, Vol. 12 No. 4, pp. 275-83.
Carpinetti, L.C.R., Gerolamo, M.C. and Dorta, M. (2000), “A conceptual framework for deployment
of strategy – related continuous improvements”, The TQM Magazine, Vol. 12 No. 5,
pp. 340-9.
Childerhouse, P. and Towill, D. (2000), “Engineering supply chains to match customer
requirements”, Logistics Information Management, Vol. 13 No. 6, pp. 337-46.
Clarke, K. (2001), “What price loyalty when a brand switch is just a click away?”, Qualitative
Market Research: An International Journal, Vol. 4 No. 3, pp. 160-8.
Croom, S. (2001), “Restructuring supply chains through information channel innovation”,
International Journal of Operations & Production Management, Vol. 21 No. 4, pp. 504-15.
Doran, D. (2001), “Synchronous supply: an automotive case study”, European Business Review,
Vol. 13 No. 2, pp. 114-20.
Doran, D. (2002), “Manufacturing for synchronous supply: a case study of Ikeda Hoover Ltd”,
Integrated Manufacturing Systems, Vol. 13 No. 1, pp. 18-24.
Evans, S. and Jukes, S. (2000), “Improving co-development through process alignment”,
International Journal of Operations & Production Management, Vol. 20 No. 8, pp. 979-88.
Gunasekaren, A., Forker, L. and Kobu, B. (2000), “Improving operation performance in a small
company: a case study”, International Journal of Operations & Production Management,
Vol. 20 No. 3, pp. 316-36.
Holweg, M. (2002), “The three day car challenge: investigating the inhibitors of responsive order
fulfilment in new vechicle systems”, International Journal of Logistics: Research and
Applications, Vol. 6 No. 3, pp. 165-183.
Hines, P. and Rich, N. (1998), “Outsourcing competitive advantage: the use of supplier
associations”, International Journal of Physical Distribution & Logistics Management,
Vol. 28 No. 7, pp. 524-46.
Hooper, M.J., Steeple, D. and Winter, C.N. (2001), “Costing customer value: an approach for the
agile enterprise”, International Journal of Operations & Production Management, Vol. 21
No. 5, pp. 630-44.
BIJ Jayaram, J., Vickery, S.K. and Droge, C. (2000), “The effects of information systems infrastructure
and process improvements on supply chain time performance”, International Journal of
13,1/2 Physical Distribution & Logistics Management, Vol. 30 No. 3, pp. 314-30.
Kumar, R. and Midha, P.S. (2001), “A QFD methodology for evaluating a company’s PDM
requirements for collaborative product development”, Industrial Management & Data
Systems, Vol. 101 No. 3, pp. 126-32.
22 Noori, H. and Lee, W.B. (2000), “Fractal manufacturing partnerships: exploring a new form of
strategic alliance between OEM’s and suppliers”, Logistics Information Management,
Vol. 13 No. 5, pp. 301-11.
Ratnasingham, P. (2001), “Inter-organisation trust in EDI adoption: the case of Ford Motor
Company and PBR Limited in Australia”, Internet Research: Electronic Networking
Applications and Policy, Vol. 11 No. 3, pp. 261-9.
Stuart, I.F. and McCutcheon, D. (1996), “Sustaining strategic alliances”, International Journal of
Operations & Production Management, Vol. 16 No. 10, pp. 5-22.
Tatikonda, L.U. and Tatikonda, R.J. (1996), “Top ten reasons your TQM effort is failing to
improve profit”, Production & Inventory Management Journal, pp. 5-9.
Wei, C-C., Lui, P-H. and Chen, C-B. (2000), “An automated system for product specification and
design”, Assembly Automation, Vol. 20 No. 3, pp. 225-33.
Womack, J.P. and Jones, D.T. (1990), The Machine that Changed the World, Rawson
Associates/Maxwell Macmillan, New York, NY/Oxford.

About the authors

David Bennett is currently a Senior Lecturer in the Technology and Operations Management
Division of Newcastle Business School, University of Northumbria. He has 25 years industrial
experience in Logistics and Supply Chain Management and he holds both a BSc and MA in
Management Science.
James O’Kane is currently Programme Director Research and Enterprise of Newcastle
Business School, University of Northumbria. He has lectured for 18 years in both Engineering
and Business fields and holds a BSc (Hons) in Physics and a PhD in manufacturing and is a
Chartered Engineer. His research interests included AI-based simulation of manufacturing
systems and enterprise systems integration and implementation. James O’Kane is the
corresponding author and can be contacted at: james.okane@unn.ac.uk

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

A benchmarking
A benchmarking methodology for methodology
metalcasting industry
Laura M.M. Ribeiro
Dept. Eng. Metalúrgica e Materiais, Fac. de Engenharia da UP, GMM/IMAT, 23
Porto, Portugal, and
J.A. Sarsfield Cabral
Dept. Eng. Mecânica e Gestão Industrial, Fac. de Engenharia da UP, Porto,
Portugal

Abstract
Purpose – To provide the basis for the development of a specific benchmarking methodology for the
foundry industry.
Design/methodology/approach – The methodology was developed with the support of the
Portuguese Foundry Association (APF). The project was structured in two stages: developing
the model for the performance assessment (including performance indicators specially designed for the
foundry companies); and conducting a pilot benchmarking exercise in order to validate the model.
Both stages received the contribution of eight relevant national foundries.
Findings – Provides information about expectations of the benchmarking partners group. Make
available a specific benchmarking model for assessing the foudries’ performance based on critical
factors. Provides foundry-oriented indicators. Confirms that benchmarking is a time-consuming tool,
demanding continuous commitment of the top-managers.
Research limitations/implications – A limitation of the study is the number of benchmarking
partners (eight foundries). The statistical analysis of the results based on this small sample was not
practicable, preventing the definition of the presumed causal relations between improvement practices
and performance outcomes.
Practical implications – The benchmarking model is now available at the APF. From now on, this
institution can provide its members with a tool for promoting the co-operation among companies and
to enhance and enlarge significantly the available information about this sector. This activity will,
certainly, help to improve organisations’ performance, upgrading the competitiveness of the
metalcasting industry in Portugal.
Originality/value – This paper helps the development of benchmarking in the metalcasting
industry in which the number of published studies is very limited.
Keywords Benchmarking, Performance measures, Quality indicators, Manufacturing systems,
Metalworking industry, Portugal
Paper type Research paper

Introduction
Benchmarking is considered one of the most effective tools of transferring knowledge Benchmarking: An International
and innovation into organisations (Spendolini, 1992; Czuchry et al., 1995). More Journal
Vol. 13 No. 1/2, 2006
importantly, when benchmarking is used to support continuous improvement pp. 23-35
strategies, it has a positive impact on competitiveness (Codling, 1998, p. 3; Karlöf and q Emerald Group Publishing Limited
1463-5771
Östblom, 1993, p. 112; Carpinetti and Melo, 2002). DOI 10.1108/14635770610644556
BIJ Benchmarking has the following characteristics:
13,1/2 .
It is a comparative study. Any benchmarking exercise is based on a structure of
qualitative and quantitative indicators to assess, comparatively, the performance
within or between organisations.
.
It can be applied to any industry or activity.
24 .
It follows an inside-out approach. Benchmarking starts with a deep understanding
of the internal processes. Then, competitors, dissimilar organisations or different
units of the same organisation are comparatively analysed.
.
It focuses on practices. The main purpose is to learn from practices that support
the best results.
.
It is a demanding methodology. The assessment of the current performance, the
searching for best practices, and the learning and adaptation are very demanding
of human resources and time.
. It is a tool for continuous learning and improvement. Practices and performance
change frequently. Therefore, organisations should adopt benchmarking as an
on-going process.

It should be noted that identifying and transferring best practices is a tricky process,
difficult to implement (Jarrar and Zairi, 2000). This is more evident when companies
are small- to medium-sized enterprises (SMEs). Typically, SMEs have severe resource
constraints, limited knowledge of benchmarking methodologies. In addition, the
organizational culture is, in many instances, not prone to change and learn.
The Portuguese foundry industry comprises 128 companies, mostly SMEs, and
includes a great diversity of products and processes. Facing global competition, the
industry needs to increase productivity. This requirement justified the launching of a
benchmarking project specially designed for foundry companies. This initiative is
expected to support improvement activities, promote business co-operation within the
industry and enlarge available business information. After a difficult process of
development of the benchmarking methodology, the Portuguese foundry industry can
benefit from a key tool to promote competitiveness.

A brief review of benchmarking methodologies

Benchmarking approaches vary both in the way the performance assessment models
are conceived, and how in practice the methodology is used. Regarding the first aspect,
there is an evident trend in developing the model according to the specific
characteristics and needs of the benchmarking partners. Referring the methodology, a
larger consensus seems to prevail: benchmarking is widely described as a structured
process following sequential steps inspired in the Deming continuous improvement
cycle: plan (plan), collect (do), analyse (check) and adapt (act).
Table I shows the activities that, according to seven different methodologies, have
to be accomplished in each stage of a benchmarking exercise. These methodologies
were selected because they are widely accepted and frequently mentioned in the
literature. The differences are more obvious between those that were designed to assist
a particular organisation, for instance, Alcoa and AT&T (Bemowski, 1991), and those
that were not developed under a specific organisation perspective (Camp, 1989;
Spendolini, 1992; Karlöf and Östblom, 1993; Codling, 1998; Andersen and Jordan, 1998).
 Plan Collect Analyse Adapt

Camp ten-steps Planning Analysis Integration

(Camp, 1989) Identify what is to be
benchmarked
Identify comparative
companies
Establish the methods for
collecting data
Alcoa’s six-steps Deciding what to benchmark
(Bemowski, 1991) Check relevancy and validity
Planning the project
Select the team
Identify the customers
Establish the scope
Select the characteristics to
be measured
Identify information sources
AT&T’s 12-steps Determine who the clients
(Bemowski, 1991) are
Train the clients in
benchmarking
Assess the clients
expectation
Determine urgency
Determine scope and type
of benchmarking needed
Select and prepare the
team
Obtain upper management
commitment
Determine current Communicate benchmark
performance gap findings and gain
Project future acceptance
performance levels Establish functional goals
Action
Develop action plans
Implement specific actions
and monitor progress
Recalibrate benchmarks
Understanding your own Learning from the data Using the findings
performance Analyse the data
Studying others Quantify the
Identify benchmarking performance gap
candidates
Prepare general and
specific questions
Prepare guidelines
regarding ethical and legal
issues
Perform the study
Develop the benchmarking Analyse the data Integrate the recommended
plan: actions
Prepare the mission Take action
statement Continue improvement
Select collection method
Develop a profile for
selecting partners
Do research about potential
partners
Document the processes
Define metrics
Set up visits and protocol
Make visits and collect data
(continued)
A benchmarking
methodology

Benchmarking
methodologies
25

Table I.

Table I.
Spendolini five-steps
(Spendolini, 1992)
Karlöf, Östblom five-stages
(Karlöf and Östblom, 1993)
Codling 3-stages 12-steps
(Codling, 1998)
Andersen five-stages
eight steps
(Andersen and Jordan, 1998)
Plan
Identify what to benchmark
Identify customers
Identify the information
requirements
Define the specific subjects
to be benchmarked
Identify the resources
required
Decide what to benchmark
Identify the client’s
information requirements
Identify the critical factors
Identify the benchmarking
partners
The planning stage
Select the subject area
Define the process
Identify potential partners
Identify data sources and
select appropriate
collection method
Plan
Determine the process to
benchmark
Understand and document
the internal processes
Measure performance of
the internal processes
Collect
Form the benchmarking
team
Define specific roles and
team members
responsibilities
Train the team members in
project management tools
Identify benchmarking
partners
Identify information sources
Gather information
Document the processes
Collect data
Find
Identify benchmarking
partners
Collect
Understand and document
the partners’ performance
and practice
26
BIJ
13,1/2
Analyse Adapt
Collect and analyse Take action
information Produce a report
Select the collection Make improvement
methods recommendations
Contact partners to Follow-up the results
approve the Continue the benchmarking
benchmarking code of process
conduct
Analyse information
Provide
recommendations for
action
Analyse Implement for effect
Identify performance Put improvements into
gaps practice
Set realistic goals
The analysis stage The action stage
Collect data and select Communicate to
partners management & others
Determine the gaps Adjust goals and develop
Establish differences improvement plan
Target future Implement
performance Review progress and
calibrate
Analyse Improve
Identify gaps in Plan the implementation of
performance and the improvements
root causes for the Implement improvements
gaps and monitor the
implementation progress
In spite of the differences, it can be concluded that all methodologies entail a sequence A benchmarking
of steps that can be applied in broad contexts. methodology
It is interesting to verify that all the methodologies recognise the importance of
achieving a thorough understanding of the internal processes before exchanging data
among benchmarking partners. They also emphasise the need of completing the
exercise through the integration of the benchmarking findings (in order to improve the
performance level). 27
However, some benchmarking methodologies differ in particular aspects. That is
the case of the Alcoa’s methodology. In fact, it does not point out the importance of the
regular replication of the benchmarking process as a support for targeting performance
indicators (Czuchry et al., 1995). Additionally, some methodologies emphasise the
planning stage (see, for instance, AT&T in Bemowski, 1991; Spendolini, 1992),
pointing out the need of:
.
assessing the users expectations about benchmarking information;
.
identifying potential barriers for the effective data exchange;
.
creating a learning environment; and
.
obtaining top management commitment and resources.

In brief, any of the methodologies referred in Table I can be applied for helping the
identification of strong and weak points. These methodologies (and many others
quoted in the bibliography) also express the simplicity of benchmarking when
regarded from the conceptual point of view. However, a common pitfall is to
underestimate the amount of internal work required by this management tool. In fact,
several benchmarking exercises did not succeed because the concerned organisations
were not prepared (lack of commitment or insufficient resources) to endure a time
consuming exercise, very demanding in terms of data gathering.

A metalcasting-oriented benchmarking methodology

Benchmarking studies are usually lead by one organisation or a group of
organisations. Independent entities (trade associations, technological centres,
universities, etc.) have also promoted benchmarking activities as a key tool for
improving their organisations performance. It is obvious that benchmarking approach
largely depends on the type of the sponsor. However, in order to be successful, the
methodology adopted (or designed) must be focused on the benchmarking partners
needs and expectations.
The benchmarking methodology discussed in this paper was developed within the
scope of the project Analysis and Selection of Benchmarking Methodologies for
Metalcasting Industry. This project, which deserved the support of the Portuguese
Foundry Association (APF), was structured in two stages:
(1) developing the assessment performance model (including performance
indicators specially designed for the foundry companies); and
(2) conducting a pilot benchmarking study in order to validate the assessment
performance model.
BIJ The methodology followed the stages of the P-D-C-A cycle. The planning stage (the
13,1/2 only one that will be addressed in this paper) entailed the sequenced steps shown in
Figure 1.
During this stage, a group of 22 pre-selected foundries were contacted by telephone
and by electronic mail. Visits were also carried out for:
.
getting acquainted with the in-house benchmarking leader;
28 .
discussing the project in more detail; and
.
evaluating expectations using a formal survey.

This procedure was time-consuming and slow: it was necessary a whole year to gain
the acceptance of eight foundries (out of the initial 22) for embarking in the
development and validation of the benchmarking methodology (steps 1 and 2).
All the in-house benchmarking leaders were at the senior manager or executive
director level. This was considered an important prerequisite for the participation in
the assessment of the model design, assuring a global vision of the company.
An expectation survey was used at step 2. The purpose of this survey was to
evaluate the primary expectations of the partners group. This survey revealed the
following priorities:
.
to grasp and to disseminate internally the benchmarking methodology
know-how;
.
to acquire a foundry-oriented assessment model;
.
to identify partners willing to share indicators and practices;
.
to get a deep knowledge on the internal processes;
.
to identify weak and strong points and improvement opportunities; and
.
to identify and adapt good practices of other companies.

1. Characterization of the framework

- deciding on the main goals
7. Correction of the model - establishing the adhesion conditions
- determining the rules for the use of the benchmarking results
ACT PLAN 2. Selection of the benchmarking partners
- promotion of upper management commitment
- identification of the in-house benchmarking leader
6. Validation of the
CHECK DO - collection and clarification of the expectations of the
assessment model benchmarking parteners
3. Design of the performance assessment model

4. Development of the performance assessment model

- identification of the key processes
ACT PLAN - identification of the critical factors
- developing specific indicators
5. Conducting of the pilot benchmarking exercise
- preparation of the survey and instructions for data collecting
Figure 1. CHECK DO
Benchmarking
methodology
Based on the experience obtained in previous empirical studies, it was decided that the A benchmarking
assessment performance model designed for the metalcasting industry should be methodology
based on three principles:
(1) To be designed for and with the benchmarking partners, in order to create a
suitable model for the specific characteristics of the steel and iron sandcasters
and aluminium alloys diecasters (so far, in Europe and particularly in Portugal,
there is no published benchmarking studies for the metalcasting industry). 29
(2) To be focused on the so-called “critical factors” concerning the manufacturing
perspective, innovation and improvement perspective and customer
perspective; the selection of these three areas was inspired in the balanced
scorecard methodology (Kaplan and Norton, 1996).
(3) To be based on a relational and hierarchical structure of indicators grouped in
three categories (Figure 2):
.
performance indicators focused on improvement practices (which are
supposed to influence the performance outcomes on critical factors);
.
performance indicators for assessing the outcomes on critical factors (which,
in turn, are supposed to explain, at least partially, the global business
performance);
.
indicators that measure global business performance (Flapper et al., 1996):
profitability, productivity, growth and customer retention.

The underlying idea is that consistent manufacturing performance is a basic condition

for the survival and success of an industrial company: understanding in full detail and
improving continuously the “manufacturing critical factors” seems to be a good and
feasible strategy. However, implementing improvement strategies that only include,
for instance, “environment”, “safety”, “innovation” and “effectiveness” (of the
manufacturing processes), does not drive, necessarily, to positive outcomes in the
long term. Other functional capacities, such as “employee motivation”, “customer
focus” and “new product development” are also crucial for the success of the
organisations and should be explored (Figure 3).
The hierarchical structure referred in the third principle can be illustrated using the
following example: in order to evaluate the environmental performance, one should
develop indicators for measuring the resources consumption, the amount of solid waste
and the amount of gaseous emissions. On the other hand, the sustainability of the
environmental performance demands a proactive management of the resources and

Global business
Improvement practices
performance

Figure 2.
Outcomes on critical
Performance assessment
factors
system
BIJ Environment

ess
13,1/2

ven

Saf
ecti

ety
Eff
Manufacturing Business results
critical factors
30
Inn

Meeting customer

at e
ov

iv ye
needs

n
Customer

ot lo
ati

io
Growth

m mp
on

retention

E
Productivity
Customer Profitability
focus

itm e r s
Ne velo
de

mm tom

t
w pm

en
Figure 3.

pro en

co cus
Performance assessment

du t

w
ct
model

Ne
waste treatment. Accordingly, improvement practices have to be identified and
evaluated: implementation of environmental management systems, reduction of
resources consumption and development of technical skills. Appropriate indicators
should be defined for measuring the implementation of these practices. The selected
indicators are summarised in Table II: “environmental management system”;
“more-clean technologies”; “reuse of wastewater”; “energy efficiency programme”;
“employees committed to environment”, “environment training time”.
The performance assessment model involves a broad and detailed structure of
indicators that supply quantitative and qualitative information related to outcomes
and practices. For each area considered (manufacturing, customer focus, and business
results) a number of performance indicators are used. Some indicators (usually
involving numeric measurements) refer the outcomes on critical factors, while others
(usually involving qualitative judgements) provide information about the performance
on practices, which help in improving those results (Table III).
The problem with the foundry-oriented process indicators is how to define them
accurately (in order to avoid different interpretations at different organisation
contexts). The following example illustrates how a promising indicator can originate
wrong results. When using the “number of new products” to evaluate the “new product
development” performance, it is common that all new references are taken into
consideration. However, using this procedure, innovative products are not
distinguished from those that result from simple modifications of the current
products. Shall a change on the size or on the coating be considered in the “number of
new products”? This issue must be discussed in detail with the benchmarking
partners. Unfortunately, this type of questions is usually disclosed in a late stage of the
benchmarking exercise.
Examples (Creese et al., 1998; Anderson et al., 1997; Business Link,
BenchmarkingIndex questionnaire www.benchmarkindex.com/ad-fs.html) from the
literature and suggestions from the benchmarking partners were used for guiding the
development and definition of the indicators (step 4). It is frequent that companies force
the inclusion of particular indicators already being used internally. This question must
A benchmarking
methodology

31

Table II.
BIJ
Critical factor: “new product development”
13,1/2
Outcome indicator: “number of new products”
Definition Number of NP launched in the last two years.
Variations in coating are not considered for
computing the number of products (new or current)
32 Data required New products (NP): number of new products sold in
the last two years; current products (CP): number of
products sold in the last two years
Formula NP *100/CP
Improvement practice indicator: “using prototyping technologies”
Definition Does your company use advanced technologies to
accelerate the development of NP and tools?
Table III. (1) Very rarely or never
Examples of quantitative (2) Occasionally, requiring external resources
and qualitative indicators (3) Regularly using CAD 3D software
for “new product (4) As (3) and using rapid prototyping and tooling
development” assessment

be handled with care. On the one hand, those indicators are already tested and familiar.
On the other hand, including them in the model tout court can lead to an excessive
number of indicators.
In the next step, the indicators’ structure was refined. The eight companies scored,
using five-point Likert scales, the envisaged indicators against two dimensions:
importance of the indicator on the performance assessment system and difficulty in
gathering the data required for computing its value. The importance scale ranges from
“not important” (1) to “highly important” (5) and the difficulty scale ranges from
“readily available” (1) to “very difficult to obtain” (5). This procedure helped in
identifying those indicators that need adjustments in order to be more suitable for the
performance assessment and to ease data collecting.
Figure 4 shows the results that came out of this procedure used on the indicators
relating to the critical factor “effectiveness of the manufacturing processes”. The

5
2 1
Outcomes’ indicators Practices’ indicators
9 5
1 Labour productivity 8 Investment on manufacturing Best choice 11
7,8 6
2 Internal scrap 9 Investment on hardware and 4 12 16 4 13
10 14 15 3
3 Rework time software for manufacturing
Importance

4 Availability 10 Supervisors skill

3
5 Breakdowns 11 Labour flexibility
6 Setup times 12 Manufacturing staff training
7 Performance 13 Self-inspection
2
14 Certified suppliers
Figure 4. Worst choice
Importance vs difficulty 15 Income deliveries on-time
(performance indicators of 16 Rejected supplies
“manufacturing 1
effectiveness”) 1 2 3 4 5
Difficulty
indicators 3, 4, 13 and 15 are the most difficult to obtain. The problem with the first one A benchmarking
is that most of the companies usually do not measure the time needed to repair castings methodology
that do not comply with quality standards. In other words, rework time is often
incorporated in the overall time of the finishing operations. The trouble with the second
indicator is the limited information available in downtime losses (for a number of
manufacturing operations).
Indicator 13 measures the proportion of the inspection time (conformance 33
inspection) spent in self-control (by the manufacturing workers). This figure is hard to
get. A more appropriated indicator is the percentage of manufacturing operations in
self-control. Finally, the data required for computing indicator 15 is frequently not
available because companies do not measure the percentage of raw materials supplied
on time (although it is very easy to get). This analysis revealed a puzzling situation,
meaning that, in practice, companies do not consider indicator 15 essential for
assessing the performance. Accordingly, it was discharged from the model.
Concluding, the development of performance indicators is a demanding activity.
This is more evident when the assessment model is designed to attain specific
benchmarking information required for different companies included in the
benchmarking group.

Pilot benchmarking exercise – a case study

The main purpose of the pilot benchmarking exercise carried out in the second stage of
this project was to test, correct and validate the initial assessment model and the
structure of indicators. The same eight foundries that collaborated previously on the
development of the performance assessment model (steps 3 and 4) were also selected
for the pilot test. A sequence of meetings hosted by APF joined all the companies
involved in order to discuss the suitability of the questionnaire used to gather the
benchmarking data (step 5).
The questionnaire is a 35-page document, divided in two parts. The first part,
describes the performance indicators. For each indicator, it is included a brief
definition, the scope, the formula and the respective unit (days/employee, e/hour,
percentage, etc.). In the second part of the document, the data required for computing
the indicators was clearly defined in order to avoid that different contexts could
generate data with different meanings. The qualitative indicators are measured using a
multiple-choice scale. In order to minimise the subjectivity associated with the
qualitative scales, a validation criteria was developed.
The data from the benchmarking questionnaires was classified according to the
foundries’ organisational and technological characteristics. The results were analysed
taking into consideration that they should be easily displayed and compared using
graphics, tables, etc. The indicators were converted in a percentile scale and each
performance score was shown in a comparative bar chart. These charts are easy to
analyse and allow the simultaneous representation of several indicators expressed in
different units.
Surprisingly, the response rate for several indicators was low. The radar chart for
effectiveness manufacturing (Figure 5) shows response rates varying between 100 and
26 percent. This result demonstrates that the ability of the companies in answering the
questionnaire differ a lot (although they were deeply involved in designing the
assessment model). This fact disclosed the difficulty associated with operational
BIJ 1 Labour productivity
13,1/2 16 Rejected supplies
100
2 Internal scrap
80
14 Certified suppliers 60
3 Rework time

40
34 12 Manufacturing staff training 20 4 Availability
0

11 Labour flexibility 5 Breakdowns

Figure 5.
Response for performance 10 Supervisors skill 6 Setup times
indicators of
“manufacturing 9 Investment/hardware, software 7 Performance
effectiveness”
8 Investment/manufacturing

(detailed) benchmarking: some time is needed before companies are able to deal with a
thorough and deep benchmarking model. Additionally, it is likely that the number of
indicators envisaged initially was excessive.
The initial performance indicators’ structure was refined once more using analytic
hierarchy process, relying again on the feedback of the participant companies. With
this methodology, the relative importance of the critical factors and the relative
priorities of the improvement practices were determined in a pairwise basis (step 7).

Summary of findings and conclusions

The final benchmarking model for assessing the foundries’ performance is now
available at the APF. From now on, this institution can provide its members with a tool
able to promote the co-operation among companies and to enhance and enlarge
significantly the available information about this sector.
In future, it is intended that APF will explore its international contacts, aiming at
exchanging benchmarking data and allowing mutual aid in the search for best
practices. This activity will, certainly, help to improve organisations’ performance,
upgrading the competitiveness of the metalcasting industry in Portugal.
The exercise confirms that benchmarking is a time-consuming tool, demanding
sound resilience and certainly a strong commitment of the top-managers. Achieving a
good agreement between partners is difficult, considering that each one wants
indicators for assessing their own specific practices and processes. It is likely to go
round several times before the assessment model begins to focus on the overall
expectations. Typically, the model evolves in a too ambitious and complex direction,
becoming unfeasible to attempt the main proposes of the benchmarking exercise.
Therefore, it is crucial to evaluate cautiously the suitability of each indicator.
Additionally, in order to prevent a premature ending of the benchmarking exercise
it is important to provide a method for organizing hierarchically the improvement
practices. In this context, analytic hierarchy process seems to be a good technique.
At the end of the pilot exercise, the authors are convinced that specific
benchmarking is a valuable, although difficult, activity.
References A benchmarking
Anderson, B., Browne, J., Devlin, J. and Rolstadås, A. (1997), “Performance measurement: the methodology
ENAPS approach”, The International Journal of Business Transformation, Vol. 1 No. 2,
pp. 73-84.
Andersen, B. and Jordan, P. (1998), “Setting up a performance benchmarking network”,
Production Planning & Control, Vol. 9 No. 1, pp. 13-19.
Bemowski, K. (1991), “The benchmarking bandwagon”, Quality Progress, Vol. 24 No. 1, pp. 19-24. 35
Camp, R.C. (1989), Benchmarking: The Search for Industry Best Practices that Lead to Superior
Performance, ASQC Quality Press, Milwaukee, WI.
Carpinetti, L.R. and Melo, A.M. (2002), “What to benchmark? A systematic approach and cases”,
Benchmarking: An International Journal, Vol. 9 No. 3, pp. 244-55.
Codling, S. (1998), Benchmarking, Gower, Aldershot.
Creese, R.C., Atluri, R. and Rammohan, N.C. (1998), “Metalcasting benchmarking”, Cost
Engineering, Vol. 40 No. 12, pp. 35-42.
Czuchry, A.J., Yasin, M.M. and Dorsh, J.J. (1995), “A review of benchmarking literature: a
proposed model for implementation”, Introduction Journal of Materials & Product
Tecnology, Vol. 10 Nos 1/2, pp. 27-45.
Flapper, S., Fortuin, L. and Stoop, P. (1996), “Towards consistent performance management
systems”, International Journal of Operations & Production Management, Vol. 16 No. 7,
pp. 27-37.
Jarrar, Y.F. and Zairi, M. (2000), “Best practice transfer for future competitiveness: a study of best
practices”, Total Quality Management, Vol. 11 Nos 4/5 & 6, pp. S734-40.
Kaplan, R. and Norton, D. (1996), The Balanced Scorecard: Translating Strategy into Action,
Harvard Business School Press, Boston, MA.
Karlöf, B. and Östblom, S. (1993), Benchmarking: A Signpost to Excellence in Quality and
Productivity, Wiley, Chichester.
Spendolini, M.J. (1992), The Benchmarking Book, AMACOM, New York, NY.

Corresponding author
Laura M.M. Ribeiro can be contacted at: lribeiro@fe.up.pt

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

BIJ
13,1/2 An investigation of supply chain
performance measurement in the
Indian automotive sector
36
Mohammed Saad
Bristol Business School, University of the West of England,
Bristol, UK, and
Bhaskar Patel
KITS, Ramtek, Nagpur, India

Abstract
Purpose – The purpose of this paper is to investigate the relevance of the concept of supply chain
performance in developing countries. The research also attempts to identify performance measure sets
for supply chain performance in the context of a developing nation.
Design/methodology/approach – The research focuses on supply chain practices in the Indian
automobile sector. It identifies and discusses the main motives and determinants for the adoption and
implementation of supply chain management concepts. It reviews the relevance of the main models to
measure the performance of supply chain in developing countries. The research is based on a
combination of qualitative and quantitative methods. Primary data were collected through
semi-structured interviews and an exploratory survey. Performance measure sets were identified
through factor analysis.
Findings – This research proposes that the concept of supply chain performance is not fully
embraced by the Indian automobile sector and highlights the difficulties associated with its
implementation.
Research limitations/implications – Further research involving other sectors and industries
needs to be undertaken in order to gain an in-depth understanding of the key factors associated with
the implementation of supply chain performance practices in India. This could also help develop a
generic model to measure supply chain performance.
Originality/value – The paper provides an attempt to adopt the concept of supply chain
performance to the Indian context and culture.
Keywords Supply chain management, Performance measures, Automotive industry,
Developing countries, India
Paper type Research paper

Benchmarking: An International
Journal
Vol. 13 No. 1/2, 2006
pp. 36-53
q Emerald Group Publishing Limited
1463-5771
DOI 10.1108/14635770610644565
1. Introduction
Businesses in many developing nations have been sheltered from competition through
protectionism at home and government intervention in foreign trades. However, rapid
deregulation, globalization of markets and gradual acceptance of competition are
making it increasingly difficult to continue to protect local markets (Madu, 1997). In
response to liberalization and globalization, business organizations in India are
striving hard to introduce new approaches such as total quality management (TQM),
just in time (JIT), business process reengineering (BPR) and supply chain management
(SCM) to enhance their performance and competitive advantage. For instance, the new
automobile policy introduced in 1993, in India had facilitated the entry of
multinationals, which have brought with them new management concepts. In terms
of SCM, this has led to significant attempts aimed at integrating the entire value chain Investigation of
with a view to enhancing the flow of goods and information from the end-user. The supply chain
passenger car segment, has for instance, undergone fundamental changes with respect
to the approach adopted to manage its supply chain. In the past few years, auto performance
assemblers have invested in their supplier base to enhance performance through
developing lasting and stable relationships.
However, there is still a lack of significant study of supply chain practices and its 37
performance in developing countries, in general and India, in particular (Austin, 1990).
There have been many attempts to measure the performance at the organizational
level, but very few at inter-organizational level. Even at the organizational level,
performance measurement is essentially focused on tangible and financial factors.
The objective of this paper is to investigate the relevance of supply chain
performance measurement in developing nations based on the case of the Indian
automobile sector. It also attempts to highlight the main enablers and inhibitors to the
implementation of both concepts, namely, SCM and performance measurement in
India. The paper comprises seven sections. Section 2 sets the context of the research,
through a review of the relevant literature and describes the methodology used by this
research. Section 3 describes the key features of the Indian context. Sections 4 and 5
highlight the key issues and challenges associated with the implementation of SCM
and how its performance is measured. Section 6 analyses the fieldwork data to assess
the implementation of SCM and its performance with respect to the Indian automotive
industry. This section also highlights the factors in the measurement of the
performance for the Indian automobile sector. The last section summarizes the key
findings and draws some conclusions related to the measurement of the performance of
supply chain in the Indian automotive sector.

2. Context, focus and methodology of the research

This research focuses on the performance measurement of supply chain within the
Indian automotive industry. As shown in Figure 1, it is underpinned by a large body of
literature which includes development and organization studies with a strong
emphasis on measuring performance at the inter-organizational level.

2.1 The Indian auto industry scenario

The Indian automotive industry is complex and different from the automotive industry in
the West. It comprises different categories: two wheelers, three wheelers, passenger cars,
light commercial vehicles (LCVs), multi-utility vehicles (MUVs) and heavy commercial
vehicles (HCVs) agricultural and farm and earth moving equipment. Some of these
manufacturers are lead players in more than one category. The Indian automotive
industry came into being in the 1950s, and has grown since under a highly regulated and
protected economic environment. The firms were subjected to strict product specific
capacity licensing, foreign collaboration, asset size and scope of industrial operations. As a
result, very few firms dominated all the products. These restrictions provided no
motivation or incentive for the firms to bring about technological improvements and new
management strategies. In addition, capacity licensing restricted Indian firms from
enjoying the scale advantages. With the new industrial policy of late-1980s and
early-1990s, the Indian government deregulated entry into the automotive industry,
dispensed with the use of licenses to control output levels, and significantly reduced
BIJ
13,1/2

38 Developing nations' industry

Indian
auto industry

Performance
measurement

Figure 1. Supply chain

Research context and management
focus

import tariffs on auto components. The liberalization of the automotive industry was also
aimed at components manufacturing sector that was reserved for small sector.
Automotive firms that came under the purview of the monopolies and restrictive trade
practices (MRTP) and foreign exchange regulation act (FERA) were permitted to enter
several areas, which only restricted to the public sector. With the end of imports restriction
aimed at reducing the outflow of scarce foreign exchange and the introduction of more
liberal import policies, importers of capital equipment are currently allotted nearly a
50 percent increase of their foreign exchange quota.
These changes led to an influx of globally competitive auto assemblers into the
Indian passenger car market. World-class firms entered this market segment, which
had only four passenger car assemblers previously. As a result of this influx of
assemblers, planned production capacity now exceeds the estimated demand for cars
in the country. Competition among assemblers has become intense, and as a
consequence firms are increasingly being innovative in order to reduce costs, enhance
quality, and improve their performance and responsiveness to customers’ demand. To
achieve these goals, existing firms as well as new entrants are all attempting to
improve their supply chains and implement lean production techniques. Indeed, there
is sufficient evidence (Hum, 1990) that local firms have been attempting to adopt
different management practices, such as TQM, BPR, JIT, SCM and other new concepts
enhance performance within and across the organizations. The majority of assemblers
in India firmly believe that SCM adaptation and implementation is a necessary if not
sufficient condition for competitiveness.
Indian automobile sector seems to be still characterized by the following factors
(Saad and Patel, 2002):
.
Indian automotive sector is fragmented with a large number of auto assemblers
in all the segments of the industry as these firms many not be able to reap the
benefits of economies of scale.
 .
The continued high levels of tariffs imposed on import products and components Investigation of
aimed at using local suppliers is hardly working as a result of their limited level
of technological capability, poor quality and lack of reliability in terms of
supply chain
delivery. performance
.
A large number of players in the automotive components sector (with over 300
small and medium sized firms servicing 24 automobile companies).
.
Around 30 percent dependency upon small-scale organizations producing and 39
selling unbranded products.
.
Only one-third of Indian auto component manufacturing companies are ISO 9000
certified.
.
Automotive component suppliers are avoiding strong dependency on assemblers
by actively dealing with the after sales.
.
In some segments auto ancillary firms have small capacities, which often lead to
shortages and lack of availability of components.
.
Integration among the partners is still in its infancy and in a majority of the cases
it is essentially vertically oriented.
.
Assemblers are still considered as the major competitive units, rather than the
complete supply chain including (at least) the first tier component suppliers.

3. Supply chain management: issues and challenges

SCM is a concept that originated in the manufacturing industries in the early-1980s. It
was developed from innovations such as JIT (Vrijhoef and Koskela, 2000) and TQM
(Wong and Fung, 1999). SCM can be seen as an example of evolutionary and
cumulative innovation, which is often described as emanating from internal programs
aimed at improving overall effectiveness (Saad et al., 2002). The focus is not only
limited to increasing the internal efficiency of organizations, but also has now been
broadened to include methods of reducing waste and adding value across the entire
supply chain (New and Ramsay, 1997; Christopher, 1998; Harland et al., 1999). SCM has
shifted the emphasis from internal structure to external linkages and processes, and is
dependent on the interaction between the organization and its external environment,
with strong feedback linkages and collective learning. It is seen as a set of practices
aimed at managing and coordinating the whole supply chain from raw material
suppliers to end customers (Harland, 1996; Vollman et al., 1997; Slack et al., 2001),
which develop greater synergy through collaboration along the whole supply chain
(Lamming, 1993; New and Ramsay, 1997).
This holistic approach is associated with the effective management of the interfaces
between all the organizations involved (Von Hipple, 1986), and the integration of both
upstream and downstream processes (Harland et al., 1999; Christopher and Juttner,
2000). This significant emphasis on co-ordination and integration is strongly linked to
the development of more effective and longer-term relationships between buyers and
suppliers (Spekman et al., 1998; Kosela, 1999).
These new types of relationships are increasingly perceived as a means to utilize
resources better through the whole supply chain (Dubois and Gadde, 2000). In addition,
they can also lead to greater transparency in transactions, increased trust and
commitment (Ali et al., 1997). There are successful examples of where SCM is
delivering significant performance improvements across the entire supply chain
BIJ (Houlihan, 1985; Burgess, 1988; Holti, 1997). It can also be an important element in
13,1/2 innovation in products, processes and organization (Edum-Fotwe et al., 2001).
Information can be more readily shared and knowledge identified, captured and
disseminated throughout the organizations in the chain (Mowery, 1988). This has led to
an increasing adoption of partnership approaches and inter-organizational alliances to
achieve significant mutual benefits involving sharing resources, information, learning
40 and other key assets (Akintoye et al., 2000).
However, SCM is a long, complex and dynamic process. Its successful
implementation needs to be associated with a thorough understanding of the
concept itself (Whipple and Frankel, 2000; Savage, 1990; Neely, 1998). Its
implementation is also seen as being closely dependent upon the ability to create,
manage and reshape relationships between individuals, organizations and networks
within the supply chain (New and Ramsay, 1997; Spekman et al., 1998; Harland et al.,
1999). It requires new organizational arrangements and culture (Neely, 1998) which
calls for considerable commitment, resources and time to develop.
It is important to recognize that SCM is complex and has proved to be difficult to
implement. It is described as a multi-factor process, reliant upon close and long-term
relationships within and between organizations (Saad et al., 2002). Its success is
associated with the challenging and difficult development of a new culture based on
shared learning, greater transparency and trust. With a greater reliance on suppliers
and the increasing emergence of outsourcing and fierce competition, the main
challenge for SCM is to sustain and continuously improve the coordination and
integration of all interactions and interfaces in order to enhance the overall
performance of the supply chain. It is, therefore important to associate the concept of
SCM based on continuous improvement with performance measurement.
However, there is also an emerging recognition that a major difficulty is associated
with what constitutes effective relationships with partners in supply chain, and how their
effectiveness can be assessed. The success of SCM is essentially reliant on less tangible
factors on which current metrics of performance are not sufficiently adequate to measure.

4. Measuring the performance of SCM

Development of the literature on performance measurements can be divided into two
distinct phases (Dixon et al., 1990). The first phase relates to the period until the 1980s
and concentrated on financial measures such as profit, return on investment and
productivity. The second phase, which commenced in the late-1980s, corresponds to
the emergence of new management concepts such as SCM. It attempts to place a
greater emphasis upon the inclusion of less tangible and non-financial measures in
performance measurements.
The shortcomings of existing systems, particularly those based on traditional cost
accounting principles, have been widely documented (Johnson and Kaplan, 1987;
Kaplan and Norton, 1992). Most of these measures focus on return on investment,
return on sales, price variances, and sales per employee and productivity (Dixon et al.,
1990). These measures lack flexibility, ignore customer requirements and place a
greater emphasis upon productivity measures. The main weakness is characterized by
a narrow or uni-dimensional focus, which does not comply with the holistic approach
of SCM and the concept of continuous improvement. This concept is at the heart of the
philosophy of performance measurement. Kaplan and Norton (1992) argue that this
difficulty can be overcome by the use of a set of measures, which can reflect this Investigation of
complexity associated with the holistic approach of SCM performance. Betchel and supply chain
Jayaram (1997) claim that integrated measures, which are cross functional have to be
used as they avoid optimization at one point in the chain without considering the performance
potential consequences at other points on the chain. Remko (1998) argues that
non-financial measures have to be developed as they supplement financial measures.
They are very crucial and include a wide range of organizational and supply chain 41
participants. For Alla et al. (1996), the major attributes of non-financial measures is
their greater emphasis on a holistic and strategic approach. They should therefore be:
.
based on organization’s strategy and needs;
.
applicable to concepts such as JIT, TQM, SCM and other approaches used by the
organizations;
.
intended for all employees;
.
lead to employee satisfaction;
.
flexible; and
. vary between locations of the organizations.

Gunasekaran et al. (2001) have suggested “time” has to be used as a strategic metric in
performance measurement. This is because controlling and compressing time shall
increase quality, reduce costs and enhance the responsiveness to customer demands
and overall productivity. However, this emphasis on time meant that efforts were not
carried out to consider the other important operational performance measures.
However, most of the current performance measures fail to clearly distinguish
various levels such as strategic, tactical and operational levels especially in a
supply chain. In this context, Gunasekaran et al. (2001) identify plan, source, make
or assemble and delivery as four links of supply chain and propose specific
metrics to measure performance at strategic, operational and tactical levels. This
proposed framework includes both financial and non-financial measures.
Benchmarking, which consists of measuring, comparing, learning and improving
(Andersen, 1998), provides a framework to evaluate both the operational and strategic
effectiveness of the supply chain (Gilmour, 1998). It is used to identify best practice and
key enablers for its effective transfer (Zaire and Whymark, 2000). It hence increases the
knowledge about SCM processes and helps quantify the performance improvement
opportunities across the whole supply chain (Stewart, 1995).
The shift to greater customized agile supply chain in late-1990s has also led to a
shift of performance metrics from stock turns and production cost in the 1980s to
customer satisfaction and value added in the late-1990s as shown in Table I.

Time period Early-1980s Late-1980s Early-1990s Late-1990s

Performance Stock turns Throughput time Market share Customer satisfaction

metrics production cost physical cost total cost value added Table I.
Shift in performance
Source: Derived from Christopher and Towill (2001) metrics
BIJ The use of non-financial metrics is crucial in measuring and fostering improvement of
13,1/2 performance of supply chains through better relationships, more shared learning and
commitment to a common purpose. The main advantages of these non-financial
measures include their flexibility and ability to take into consideration soft aspects
linked to empowerment, trust and continuous improvement. However, this can lead to
a wide range of measures, which can be difficult to select, adopt, monitor and manage
42 particularly in the context of developing countries.
One of the most important and relevant models to measure the performance of SCM
is the European Foundation for Quality Management’s Business Excellence Model.
This model comprises two distinct subsets of performance factors, broadly classified
as enablers and results; and can accommodate the complexity, the dynamics and
interactions that feature SCM. Both enablers and results can include financial and
non-financial metrics, which can enable managers to monitor, measure and foster the
improvement of performance based on hard and soft factors. It is also contingent to the
context used and can, to some extent, be used in the context of developing countries.
The theory underpinning the Business Excellence Model is that the enablers are the
levers that management can pull to deliver future results such as those needed for
greater empowerment, motivation, trust and learning. However, Neely et al. (2000)
argue that it can be difficult to operationalise. The terms used in this model are so open
and can be interpreted in so many ways, that any single organization could decide to
capture any one of several dozen different measures of performance under each of the
headings.
A major challenge is to identify, evaluate and select the parameters or metrics,
which are appropriate to assess performance. The process of deciding which measures
of business performance to adopt is a valuable one, not least because it forces
management teams to be very explicit about their performance priorities and the
relationship between them, thereby exposing, and offering an opportunity to resolve,
any hidden differences of opinion. Although such frameworks are undoubtedly
valuable, their adoption is often constrained by the fact that they are to be used in an
altogether different business operating environment in which measures of performance
might be useful in the context of developing nations, but provide little guidance on how
the appropriate measures can be identified, introduced and ultimately used to manage
the business. Another major difficulty relevant to the concept of SCM and its
performance measurement is the limited availability of literature and research on the
application of these concepts in the context of developing nations. And as the issues
related to purchasing practices, number of tiers in the supply chain and strategies vary
from one country to another; performance measures are bound to reflect these
differences. Hence, the need for organizations in developing nations such as India to
adopt or develop a framework suitable to their needs and context taking into account,
indigenous economic, cultural, social and political factors. Table II identifies and
proposes a summary of key features of the main models of performance measurement
and their relevance to the Indian supply chain.

5. Methodology
The choice of the combined qualitative and quantitative methodology is essentially
motivated by the need to gain an insight into the implementation and the relevance of
the concepts of SCM and performance measurement in the Indian automotive industry.
Framework/models Key features Relevance to SCM Relevance the Indian context

Frame work for Three main areas of R& D, operations Could reflect the dynamics aspect of SCM and Could take into consideration the
time-based and sales and marketing are identified the contingency of the context in which SCM is specific aspect of the context of
competition Azzone where time measures could be used adopted developing countries. However, very
et al. Identifies appropriate metrics for Emphasize on time and do not consider other few organizations are pursuing the
(1991) performance measurement operational performance measures such as time-based competitive strategy in
quality, coat and delivery question
Balance score card Identifies financial, customer, internal Could be appropriate to reflect the holistic Could be very useful tool when greater
Kaplan business, innovation and learning approach of SCM and the importance of importance is given to empowerment
and Norton (1992) perspectives non-financial measures, soft issues and and when businesses and processes are
Integrating different classes of business, continuous improvement fully integrated along the value chain
financial non-financial, internal and Very useful tool at the top management level,
external measures but it’s applicability at shop floor level needs to
be further investigated
Frame work by Two types of performance measures Attempt to go beyond the end result and to Perceived as difficult to identify the
Fitzgerald et al. related to results and determinants of identify and measure the main determinants of determinants of the end result and not
(1991) results results. In addition, this model is essentially appropriate to manufacturing, where
geared towards the needs of the service sector the emphasis is still placed on hard
factors and end results
Brown’s (1996) Identifies differences among the input, Serious attempt to introduce a holistic approach Perceived as difficult to use as the
framework process and output measures and to measure the three main elements of the information flows in the supply chain
promotes a more horizontal and operations process (input, transformation and still follow a hierarchical pattern. This
comprehensive approach to the output) but essentially through the use of a impedes a comprehensive
management and measurement of quantitative approach understanding of the whole operations
operations and processes process and the adoption of an
affective holistic approach
ICAS (2000) Provides very detailed list of financial Could be useful as it enables to incorporate the Could be very appropriate to SCM.
and non-financial performance soft issues of SCM. A serious attempt to However, there is still a significant
measures integrate all factors to measure the whole dependency on hard and financial
performance performance measures
(continued)
performance
supply chain

automobile sector
metrics to the Indian
performance models and
Relevance of SCM
43

Table II.
Investigation of

Table II.
Framework/models Key features
SMART (2000) Defines hierarchical levels of objectives
(strategic and measures: strategy at the top,
measurement followed by objectives for each business
analysis and unit, at the next level objectives for
business operating systems and the
reporting technique
(1988-1989)) fourth level is represented by customer
satisfaction, productivity, etc.
PMQ (performance Emphasize on identifying performance
measurement improvement areas
questionnaire) Questionnaire structure for identifying
by Dixon et al. performance measures
(1990)
Framework by Identifies use of resources, desired out
Benita (1999) put and flexibility as components of
effective supply chain performance
measurement. It provides a quantitative
approach for flexibility measurement. It
provides a detailed list of performance
measures
Frame work by It identifies four links of supply chain:
Gunasekarn et al. plan, source, make, and deliver. It
(2001) provides a detailed list of metrics for
strategic, operational and tactical level
Relevance to SCM
SMART is very appropriate as the approach is
highly integrated; however, the emphasis on
measures on quality, and delivery links are not
explicit. This acts as a hindrance for continuous
improvement
Could be used as it provides a framework for
identifying improvement areas, but it does not
identify the links among various improvement
areas
Could be very effective for supply chain
performance as it takes in to account flexibility
(ability of the system to respond to
uncertainties). The framework does not take in
to account the soft issues of supply chain
It incorporates issues such as source
performance and customer satisfaction. It could
be very useful in performance evaluation of
supply chains
44
BIJ
13,1/2
Relevance the Indian context
The issue of quality is a critical issue in
the context of the research. It can be
appropriate to the Indian context to
help improve quality. It is perceived as
easy to implement as it is driven by the
top
It requires a culture of empowerment
and learning
As this frame work does not address
the issues related to supplier relations
and supply performance it may not be
useful to adopt in an integrated supply
chain performance measurement
The measures are identified from
literature and there are to be tested for
the context of application. Specific
strategic metrics (such as buyer
supplier partnership level) cannot be
used, as the developments are not on
par with the SCM practices in
developed nations. The limited
understanding of where value is added
and the of customer focus culture can
impede the adoption of this model
The methods to collect data include desk research, interviews and questionnaire Investigation of
survey. supply chain
In addition to the review of the academic literature, this research is also based on
the secondary data provided by the Automotive Components Manufacturers performance
Association of India (ACMA), Society of Indian Automotive Manufacturers (SIAM)
and the policy documents published by the Indian Ministry of Commerce and
Industry (2002). 45
The fieldwork was carried out in two stages. The first stage was carried through
semi-structured interviews of senior mangers of Indian auto assembler and component
manufacturers. The main objectives of the first stage were to ascertain the issues
pertaining to practices of supply chain in the Indian automotive industry and to
investigate the main determinants associated with its implementation.
A total of 14 top managers in charge of operations and supply chain were
interviewed. The interviewees were drawn from four major auto assemblers and two
auto component manufacturers. The duration of the interviews varied from 90 to
120 min. The composition of samples for both the interviews and the questionnaires
were suggested by the SIAM and ACMA. In both cases the samples were formed by
top managers in charge of operations and supply chain.
The second stage of the fieldwork was based on an exploratory questionnaire
survey and was divided into two main parts. The first part was essentially focused on
issues related to the implementation of SCM in India such as the motives, enablers and
inhibitors, supplier selection, supplier relationships and outsourcing. The second part
focused on the concept of performance measurement which includes performance
frameworks, performance measures/metrics and issues of application and adaptability
of European Foundation Quality Management (EFQM), balance score card and supply
chain operations reference (SCOR) models.
Performance measures highlighted by the literature were also checked in order to
gain a better understanding of their relevance and applicability in the Indian context.
These performance measures and metrics are used in the factor analysis and shown in
Figure 2.

Transportation Time

Buyer supplier
Manufacturing Customer relation
and Inventory satisfaction management

Figure 2.
Financial efficiency Information Performance “measure
management sets”
BIJ The questionnaire was piloted with three subject experts before it was finally
13,1/2 administered. This was done to cross verify the contents, structure and nature of the
questions asked in the questionnaire and improve validity (Mitchell, 1996).
The sample for this questionnaire was drawn from 14 Indian auto assemblers
(passenger car segment) organizations and from 150 auto component manufactures
(organized sector). During the pilot testing, it was suggested to consider two
46 respondents from each of the assembler organizations in order to make the sample
as representative as possible. Therefore, 28 members of the sample were identified
from all the 14 auto assembler organizations. This sample selection replicates the
convenience and judgmental sampling methods (Barnett, 1991).
In case of the component manufacturers, as they were spread throughout the
country, the sample included four managers from each of the four regions (Eastern,
Western, Southern and Northern regions). This selection was essentially justified by
quota sampling in which selection of cases within strata is entirely non-random
(Barnett, 1991).
The overall sample comprised 44 managers of which 40 have responded. This
can be seen as the result of one of the researchers meeting and briefing all
members of the sample and the help offered by ACMA and SIAM.
In order to validate the sample for the factor analysis, the second part of
questionnaire had to be sent by email to an additional 30 managers of which 24 have
completed and returned the form.

6. The key features of SCM performance in the Indian automotive sector

6.1 Motives for SCM adoption
The motives for adopting SCM by the Indian auto organizations, as shown in Figure 3,
clearly indicate that cost competitiveness, reduction in cycle time and inventory
management score high. The weighted scores of these motives are 4.82, 4.44, and 4.25,

5.0
4.5
4.0
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0
competitiveness

Information and

transportation

Better quality

outsourcing

supplier base

organizational
communication

Reduction in

Flat and lean

Rationalize
Effective
and control

structure
costs
tech.
Cost

Figure 3.
Motives for SCM adoption
respectively, on a scale of maximum 5 (with a Cronbach’s a value of 0.7082). Customer Investigation of
focus and quality enhancement are also listed as important motives. This clear supply chain
indicates that the adoption of SCM is significantly linked to cost control including
transportation cost. It is also interesting to note that respondents have not considered a performance
flat and lean organization structure as an important motive (weighted mean value of
2.5). This suggests that structures of most Indian organizations are still of a traditional
and hierarchical type. It can also be interpreted as a lack of the understanding of the 47
pre-requisites of SCM. Similarly, rationalizing the supplier base is not rated as an
important motive. This contradicts the philosophy of SCM based on the rationalization
of the supply base. Indian firms tend to associate a large supply base with
competitiveness based on low cost.

6.2 Main enablers and inhibitors

The main enablers highlighted by the desk research include:
.
new economic and industrial policies liberalizing the market and allowing
multinational enterprises to work in partnership with Indian organization;
.
improved infrastructure and help provided by the Indian government;
. quality of skilled and cost effective workforce; and
.
greater emphasis on learning and innovation.
However, the findings from the interviews and questionnaires suggest that most of the
above factors impede rather than foster implementation of SCM. For 88.2 percent of
the respondents, the state support in terms of policy reforms is insufficient.
The infrastructure (including energy, transportation and ICT) is also perceived as
inadequate by 80 percent of the respondents.
The respondents have also identified organizational factors such as structure and
management styles (88 percent) and lack of professional purchasing practices
(75 percent) as the major obstacles to the implementation of SCM (Figure 4). These
responses are clear indication that the Indian auto industry does not fulfill the
necessary requirements for a successful implementation of SCM. In addition, many
managers are still operating under “command-and-control” principles. Most

100
90
80
70
60
50
40
30
20
10
0
huge number

capacity of

infrastructure

Command and

empowerment

participation

professional
purchaisng
of suppliers

management
suppliers

Lack of IT

employee
smaller

employee

Lack of

lack of
Lack of
control

Figure 4.
Factors affecting the
implementation of SCM
BIJ organizations are very hierarchical with rigid bureaucratic procedures and a strong
13,1/2 public sector culture. The strong boundaries within and between organizations are
significantly impeding the development of cross-functional teams, communication
needed for a greater trust, participation, empowerment and continuous improvement.
As already mentioned, large supply base and of quality suppliers are not identified
as major impeding factors. This reflects a lack of clear understanding of SCM and its
48 implementation.

6.3 Supplier selection

The key factors to select suppliers identified by most of the respondents (Figure 5):
.
supply delivery lead time (96 percent);
.
historical rejection rate (90 percent);
.
geographical proximity (88 percent); and
.
reliability (86 percent).

This suggests that the selection of suppliers is essentially motivated by the need to
address the main inhibitors that most of the managers are experiencing in managing
their supply chain on a day-to-basis. As suggested by Carter and Narasimhan (1990),
one of the main selection criteria is related to the experience and financial strength
and technical capabilities of suppliers. There is clear evidence of a strong adoption of
“localisation strategy” by some of the assemblers. This strategy is aimed at
encouraging suppliers to be located near them in order to reduce the uncertainties and
risk associated with deliveries and inventories.
However, flexibility of supplier (with only 27 percent of respondents) is not
considered as an important criterion for the selection of suppliers. Long-term
relationships with suppliers are only supported by 62 percent. Similarly cultural
compatibility with suppliers is identified as an important criterion by 56 percent. This

100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
perspective

effectiveness

Quality of delivery

Supplier's ability

Supplier delivery

Supplier capacity

Flexibility

compatibility
Reliability

Geographical

rejection rates
Long term

(delivery)
to cost saving

proximity

Historical
Cultural
lead times
initiatives
Cost

Figure 5.
Supplier selection criteria
suggests that most of the key features and pre-requisites of SCM are either not Investigation of
understood or simply ignored. supply chain
6.4 Supply chain performance
performance
Although Indian organizations have realized the importance of performance
measurement of supply chain, they are predominantly using financial,
productivity-based measures and less of the intangible or soft measures, as 49
observed in the case of 88 percent of the respondents. As already discussed, the
emphasis remains upon productivity and cost related measures.
Models such as balance score card, EFQM, ICAS, SCOR and others which
incorporate intangible measures and soft aspects linked to empowerment, learning and
continuous improvement may be difficult to use in the Indian environment. Most of the
respondents have identified the following obstacles to the use of the above models:
.
difficulty to operationalise;
.
specificity of the Indian business and operating environment;
.
management practices; and
.
culture.

In addition, in most cases the efforts to measure performance were confined to

organizational boundaries rather the whole supply chain.

6.5 A proposed set of performance measures

Part of the field work was also aimed at investigating performance measures
appropriate to the Indian automotive context. Managers were asked to rank the various
measures of performance on 1-7 Lickert scale (7 being extremely important measure
and 1 extremely unimportant).
The resulting ranked data have been analyzed using the factor analysis technique,
to simplify the data and identify major sets of performance measures. This was carried
out using the 11.1.0 version of SPSS. The “eigen value” criterion is used for extracting
factors. All the variables were categorized into seven factors. These seven factors
together explain 78.9 percent of the total variance. The output of factor analysis
suggested all variables could be grouped into seven groups/measure sets. These
groups have been named and are as shown in Figure 2. They consist of tangible
measures such as cost and productivity and intangible measures such as
communication, learning and trust. The inter-links indicate interdependency of these
“measure sets” and also highlight the necessity of usage of these “measure sets” in
conjunction with all other measures. It is clearly evident that transportation and
information management has predominant influence on the performance of supply
chain in the Indian context. In addition to the costs of inbound and outbound
movements, the cost of information sharing, management and transfer are also
significant.
The composition and combination of measures from each of these measure sets is
contingent upon each sector and/or organization. This can help define and identify
appropriate measure sets, which can be more compatible with the Indian context.
These “measure sets” can facilitate the effective adoption of supply performance
measurement in contexts such ad the Indian automotive sector.
BIJ 7. Discussion
13,1/2 The main points arising from the literature review and findings from the field work are
summarized in Table II which attempts to highlight the relevance and the applicability
of different performance frameworks and measures for supply chain in the Indian
context.

50
7.1 Conclusion
There is a clear evidence from the desk research and the field work of significant
changes within both the internal and external environment of Indian companies
resulting from the introduction of new economic and industrial policies. These policies
are helping to create a more liberalized and open economy and fostering
competitiveness through learning and innovation. There is an increasing awareness
about the need to collaborate with world-class players and enhance performance
through the use of new management concepts. Indian companies are increasingly
attempting to improve the coordination and integration with their suppliers both
within and outside the national boundaries.
However, most innovations are influenced by improving the most tangible factors
which are easy to measure such as cost and productivity. Improving and establishing
long-term relationships are also not fully perceived as important. Similarly, there is
reluctance to adopt flatter and less hierarchical organizational structures. Learning is
important but essentially aimed at the technical capabilities, which is perceived as
tangible and easily measurable. This suggests that there is an awareness about the
need to measure and continuously improve performance. However, this is essentially
carried out through traditional models of performance measurement based on tangible
factors.
There is also a tendency to believe that key factors such as quality, delivery and
lead time can be improved by selecting the suppliers who possess significant technical
experience and expertise. Performance improvement is essentially focused on the
acquisition of technical and tangible factors. It also suggests that performance
improvement is not seen as a common task based on shared learning and joint problem
solving.
Although this paper has placed a greater emphasis on the introduction of supply
chains rather than on performance measurement, it has clearly shown that the Indian
automotive sector is not embracing the whole philosophy of supply chain performance
as highlighted in the literature review. This can be interpreted as either a lack of proper
understanding of the concept or an interesting attempt to adapt the concept to the
Indian context and culture. Further research is needed to prove this either way.

References
Akintoye, A., McIntosh, G. and Fitzgerald, E. (2000), “A survey of supply chain collaboration and
management in the UK construction industry. Supply chain management in construction –
special issue”, European Journal of Purchasing and Supply Management, Vol. 6,
pp. 159-168.
Alaa, M., Ghalayini and James, S.N. (1996), “The changing basis of performance measurement”,
International Journal of Operations & Production Management, Vol. 16 No. 8, pp. 63-80.
Ali, F., Smith, G. and Saker, J. (1997), “Developing buyer-supplier relationships in the automobile Investigation of
industry, a study of Jaguar and Nippondenson”, European Journal of Purchasing and
Supply Management, Vol. 3 No. 1, pp. 33-42. supply chain
Andersen, B., Randmael, S. and Fagerhaug, T. (1998), final report from the bench marking performance
activity, SMARTMAN document, Trondheim.
Austin, J.E. (1990), Managing in Developing Countries, Free Press, New York, NY.
Azzone, G., Maseela, C. and Bertele, U. (1991), “Design of performance measures for time based 51
companies”, International Journal of Operations & Productions Management, Vol. 11 No. 3,
pp. 77-85.
Barnett, V. (1991), Sample Survey Principles and Methods, Edward Arnold, London.
Benita, B.M. (1999), “Measuring supply chain performance”, International Journal of Operations
& Productions Management, Vol. 19 No. 3, pp. 275-92.
Betchel, C. and Jayaram, J. (1997), “Supply chain management: a strategic perspective”,
International Journal of Logistics Management, Vol. 8 No. 1, pp. 15-34.
Brown, M.G. (1996), Keeping Score: Using the Right Metrics to Drive World-Class Performance,
Quality Resources, New York, NY.
Burgess, R. (1988), “Avoiding supply chain management failure: lessons from business process
reengineering”, International Journal of Logistics Management, Vol. 9, pp. 15-23.
Carter, J.R. and Narasimhan, R. (1990), “Purchasing in international market place: implications
for operations”, Journal of Purchasing & Materials Management, Vol. 26 No. 3, pp. 2-11.
Christopher, M. (1998), Logistics and Supply Chain Management – Strategies for Reducing Cost
and Improving Service, Financial Times, Prentice-Hall, Englewood Cliffs, NJ.
Christopher, M. and Juttner, U. (2000), “Developing strategic partnerships in the supply chain: a
practitioner perspective”, European Journal of Purchasing & Supply Chain Management,
Vol. 2 No. 6, pp. 117-27.
Christopher, M. and Towill (2001), “An integrated model for the design of agile supply chains”,
International Journal of Physical Distribution & Logistics Management, Vol. 31 No. 1,
pp. 235-46.
Dixon, J.R., Nanni, A.J. and Vollman, T.E. (1990), The New Performance Challenge; Measuring
Operations for World Calls Competition, Dow-Jones Irwin, Homewood, IL.
Indian Ministry of Commerce and Industry (2002), documents and reports 1997-2002 on Indian
automobile industry provided by ACMA and SIAM.
Dubois, A. and Gadde, L. (2000), “Supply strategy and network effects – purchasing behaviour in
the construction industry”, European Journal of Purchasing & Supply Chain Management,
Supply Chain Management in Construction – special issue, No. 6, pp. 207-15.
Edum-Fotwe, F.T., Thorpe, A. and McCaffer, R. (2001), “Information procurement practices of
key actors in construction supply chains”, European Journal of Purchasing & Supply
Management, No. 7, pp. 155-64.
Fitzgerald, L., Johnston, R., Brignall, T.J., Silvestro, R. and Voss, C. (1991), Performance
Measurement in Service Businesses, The Chartered Institute of Management Accountants,
London.
Gilmour, P. (1998), “Bechmarking supply chain operations”, Bechmarking for Quality
Management and Technology, Vol. 5 No. 4, pp. 283-90.
Gunasekaran, A., Patel, C. and Tirtirogulu, E. (2001), “Performance measures and metrics in
supply chain environment”, International Journal of Operations & Production
Management, Vol. 21 Nos 1/2, pp. 71-87.
BIJ Harland, C.M. (1996), “Supply chain management relationships”, Chains, Networks, British
Academy of Management, Vol. 7.
13,1/2
Harland, C.M., Lamming, R.C. and Cousins, P.D. (1999), “Developing the concept of supply
strategy”, International Journal of Operations & Production Management, Vol. 19 No. 7,
pp. 650-73.
Holti, R. (1997), “Adapting supply chain for construction”, Workshop Report CPN727,
52 Construction Productivity Network, CIRIA.
Houlihan, J.B. (1985), “International supply chain management”, International Journal of Physical
Distribution & Materials Management, Vol. 3 No. 4, pp. 22-8.
Hum, S. (1990), “Industrial progress and the strategic significance of JIT and TQC for developing
countries”, International Journal of Operations & Production Management, Vol. 11 No. 5.
Johnson, H.T. and Kaplan, R.S. (1987), Relevance Lost: The Rise and Fall of Management
Accounting, Harvard Business School Press, Boston, MA.
Kaplan, R.S. and Norton, D.P. (1992), “The balance score card; measures that drive performance”,
Harvard Business Review, pp. 71-9.
Kosela, L. (1999), “Management of production construction: a theoretical view”, Proceedings of
the 7th Annual Conference of the International Group for Lean Construction IGLC-7,
Berkeley, pp. 241-52.
Lamming, R. (1993), Beyond Partnership: Strategies for Innovation and Lean Supply,
Prentice-Hall, New York, NY.
Madu, C.N. (1997), “Quality management in developing economies”, International Journal of
Quality Science, Vol. 2 No. 4, pp. 272-91.
Mitchell, V. (1996), “Assessing the reliability and validity of the questionnaires: an empirical
example”, Journal of Applied Management Studies, Vol. 5 No. 2, pp. 199-207.
Mowery, D.C. (1988), International Collaborative Ventures in US Manufacturing, Ballinger,
Cambridge, MA.
Neely, A. (1998), Measuring Business Performance – Why, What and How, Economist Books,
London.
Neely, A., Bourne, M. and Kennerly, M. (2000), “Performance measurement system design”,
International Journal of Operations & Productions Management, Vol. 20 No. 10.
New, S. and Ramsay, J. (1997), “A critical appraisal of aspects of the lean approach”, European
Journal of Purchasing & Supply Management, Vol. 3 No. 2, pp. 93-102.
Remko, I.V.H. (1998), “Measuring and improving performance in the supply chain”, Supply Chain
Management, Vol. 3 No. 4, pp. 187-92.
Saad, M., Jones, M. and James, P. (2002), “A review of the progress towards the adoption of
supply chain management (SCM) relationships in construction”, European Journal of
Purchasing & Supply Management, Vol. 8, pp. 173-83.
Saad, M. and Patel, P. (2002), “Issues and challenges in the supply chain performance
measurement in developing nations: a case of Indian automobile sector”, EUROMA 2002
Conference Proceedings, Vol. 2, pp. 1429-40.
Savage, C. (1990), Fifth Generation Management, Digital Press, Burlington, MA.
Slack, N., Chambers, S. and Johnston, R. (2001), Operations Management, 3rd ed., Financial
Times, Prentice-Hall, Eaglewood cliff, NJ.
Spekman, R.E., Kamauff, J.W. Jr and Myhr, N. (1998), “An empirical investigation into supply
chain management: a perspective on partnerships”, International Journal of Physical
Distribution & Logistics Management, Vol. 28 No. 8, pp. 630-50.
Stewart, G. (1995), “Supply chain performance benchmarking study reveals keys to supply chain Investigation of
excellence”, Logistics Information Management, Vol. 8 No. 2, pp. 38-44.
Vollman, T., Cordon, C. and Raabe, H. (1997), Supply Chain Management in Mastering
supply chain
Management, FT Pitman, London, pp. 316-22. performance
Von Hipple, E. (1986), “Co-operation between rivals: informal know how trading”, Research
Policy, Vol. 16 No. 5, pp. 21-8.
Vrijhoef, R. and Koskela, L. (2000), “The four roles of supply chain management in construction”, 53
European Journal of Purchasing & Supply Management, Supply Chain Management in
Construction – special issue, Vol. 6, pp. 169-78.
Whipple, J.M. and Frankel, R. (2000), “Strategic alliance success factors”, The Journal of Supply
Chain Management, Vol. 36 No. 3, pp. 21-8.
Wong, A. and Fung, P. (1999), “Total quality management in the construction industry in Hong
Kong: a supply chain management perspective”, Total Quality Management, Vol. 10 No. 2,
pp. 199-208.
Zaire, M. and Whymark, J. (2000), “The transfer of best practice: how to build a culture of
benchmarking and continuous learning – part 1”, Benchmarking an International Journal,
Vol. 7 No. 1, pp. 62-79.

Further reading
Blackburn, J.D. (1991), Time-Based Competition: The Next Battleground in American
Manufacturing, Business One Irwin, Homewood, IL.

Corresponding author
Mohammed Saad can be contacted at: mohammed.saad@uwe.ac.uk

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

BIJ
13,1/2 Best practices of collaboration
between university and industrial
SMEs
54
P. Peças and E. Henriques
Technology and Management Group, Department of Mechanical Engineering,
Instituto Superior Técnico, Lisboa, Portugal

Abstract
Purpose – The purpose of this paper is to contribute to the implementation of best practices of
collaboration between university and industrial small- and medium-sized enterprises (SMEs).
Design/methodology/approach – This paper presents the experience carried on by a university
group fostering the collaboration with SME companies involving young engineering students and
researchers in projects designed for the resolution of real industrial problems. A collaboration model is
proposed and described. Four real case studies are presented.
Findings – The purposed model promotes the involvement of the young engineers with authentic
industrial experiences, enables the build-up of their practical framework and encourages their
entrepreneurial growth. It also promotes the innovation process in SME companies through the close
collaboration with universities.
Practical implications – The collaboration between universities and SME companies should be
based on a small projects base. These projects must be focus in localized and specific problematic
areas in the industrial companies, where the potential of improvement and innovation is large, must
diagnose the problematic situation and propose new and efficient solutions supported by
technical/scientific methodologies. The involvement of managers and collaborators of SME
companies and the clearly definition of their roles in the project are fundamental issues for the
collaboration success.
Originality/value – The model presented in this paper describes an innovative step-by-step
procedure, easy to be implemented by the universities. It emphasises the impact of some details during
the collaboration process that enhance the success potential of university-SME companies’ projects,
the quality of the research work produced and the quality of young future engineers training. It also
fits with the SME companies’ demands of a mix of structured knowledge and empirical experience.
Keywords Universities, Small to medium-sized enterprises, Lean production, Best practice, Partnership,
Portugal
Paper type Research paper

Benchmarking: An International
Journal
Vol. 13 No. 1/2, 2006
pp. 54-67
q Emerald Group Publishing Limited
1463-5771
DOI 10.1108/14635770610644574
Introduction
Manufacturing has changed radically over the course of the last 20 years. Moreover,
the only thing certain in the future is the change acceleration in the manufacturing
domains. New products and processes are and will be foster by the emergence of new
manufacturing technologies, stimulated by intense competition. As complement, new
management and labour practices, organizational structures and decision-making
methods will emerge (NRC, 1998; Tavares, 2000). With regard to recent studies
concerning the manufacturing challenges for the following decades a critical step will
be the development of an underlying technical foundation through research performed
by industry, academia, and government institutions (NRC, 1998; Mateus, 2000). Taking
advantage of the synergies of these three research actors, manufacturers will fulfil
the gap between current practices and the vision of manufacturing in 2020 by University and
achieving concurrence in all operations, by integrating human and technical resources industrial SMEs
to enhance work-force performance and satisfaction and by “instantaneously”
transform information gathered from a vast array of sources into useful knowledge to collaboration
support effective decision-making processes (Alazmi and Zairi, 2003; Cahill, 2001).
Several authors identified the existence of barriers on the way to accomplish the
referred challenges. Despite the dynamic increasing of the technological innovation 55
processes between academia and industry the revenues of this collaboration are still
very small relative to university research budgets (Graff et al., 2002). Following
Harmon et al. (1997), university transfer agencies intend to facilitate technology
transfer but they give more emphasis on disseminating resources and competences
rather than providing assistance in network building and relationship marketing
efforts. In accordance with the formers Murphy (1993) states that the expansion of
higher education has a low impact in the countries economy due mainly to the
inadequacy of the technology transfer policies/methodologies. Liyanage and Mitchell
(1994) refer that collaboration between academia and industry remains dependent
upon the cultural, organizational and management characteristics of the partners
engaged in the cooperative activity. Therefore, the strategies used to establish strong
and long-term relationships between the university and industry must be adapted to
the intrinsic characteristics of the organizations involved in the process. Santoro and
Chakrabarti (2001) identified clear differences between the strategies that are used by
the large companies to perform research and development projects and the strategies
that must be used to promote the involvement of traditional small- and medium-sized
enterprises (SMEs) in the innovation process.
With a core of highly skilled personnel, the large manufacturing organizations have
the ability to understand the potential interest of the resources and competences
disseminated by the university agencies (Santoro and Chakrabarti (2001); Harmon et al.,
1997). Additionally, the building of huge projects is facilitated either through financing
by governmental institutions because of their social and economical impacts, or
through the involvement of large academia means – the last ones always seeking basic
research projects for papers publication (Todd et al., 2001). Furthermore, these
organizations possess strong internal research and innovation groups, so it is expected
that will overcome the referred challenges (Mesquita et al., 2000).
Nevertheless, for SMEs it will be difficult to overcome those huge challenges mainly
due to the lack of internal human resources and to its traditional culture of poor
investment in education and training (Matlay and Hyland, 1999). The SMEs have
specific needs, usually more technological and/or organisational based (Mesquita et al.,
2000). So, SME are manly interested in using their relationships with university to
address specific needs, which are nuclear to their business performance (Santoro and
Chakrabarti, 2001). Actually, the few successful cases involving SMEs in innovation
processes with academia are based on short-term agreements and involve small
amounts of financial resources. Such relationships are generally driven to support
small risk applied research with results that can be exploited in the immediate future
(Mesquita et al., 2000; Tavares, 2000). The university strategy fostering the
involvement of SMEs with academia must have an interpersonal approach rather than
formal (Demain, 2001). Also, SME owners and managers must be aware that the
relationship with the university community is an important skill that they need to
BIJ possess to enhance the opportunities to initiate a collaborative research project
13,1/2 (Harmon et al., 1997).
Keeble et al. (1999) give a strong emphasis on the SME-university networking and
collaborative research to the collective and dynamic learning process of localized
industrial regions. Accordingly, Jones-Evans et al. (1999) refer that this development
could introduce particular benefits in small peripheral economies, which have
56 demonstrated low levels of indigenous industrial technological development. As a
result, universities could become increasingly important for the development of the
local economies in smaller countries on the periphery of Europe.
Therefore, in a peripheral country like Portugal, where the manufacturing industry
framework is comprised mainly by SME and where the number of micro companies is
very significant (Mateus, 2000), a strong impulse in the innovation process is a key
factor for the increase of competitiveness (Mesquita et al., 2000).

Research methodology
The Industrial Management Group of the Mechanical Department of Instituto Superior
Técnico (Technical University of Lisbon, Portugal) has developed intense partnerships
with Portuguese manufacturing companies. Beside other activities, since 1998 this
group has carried out more than 20 short duration projects with Portuguese SMEs.
These projects with duration between 6 and 12 months were run by students of the
final year of the engineering courses. The basic aim of the projects was to increase
the productivity and competitiveness of the manufacturing companies through the
implementation of dedicated “simple” engineering-based solutions.
Several approaches were used as regards to the projects management and to the
methodologies for successful interaction between the companies and the university.
Among the referred set of projects and mainly for the first ones some unsuccessful
occurred in what concerns the fulfilment of the original objectives. The causes for these
failures were identified, analysed and crosschecked with the successful projects. The
approach has been optimised through an error-learning process, originating the increase
of the success degree with the number of projects accomplished. The degree of success
was measured qualitatively based mainly on the company involvement level, number of
developed engineering-based solutions that achieved an implementation stage in the
industrial environment and company requests for new project/collaboration.
The result of this process is the model described in this paper. In order to foster the
validation of the model different projects developed in several industrial sectors are
also presented. The described projects originated significant productivity
improvements through the application of methodologies like SMED[1], discrete-event
simulation and production re-engineering, among others. One must refer that a
considerable number of projects have been extended and continue in a collaboration
framework between academia, final year engineering students and industry, with
results that are over the initial objective.

Model scope
In these short-duration projects the students must focus on a company critical problem.
The students perform the diagnosis, followed by the development of potential solutions
and impact estimation. The company collaborators, including management team, must
be involved on the process from the early beginning. This practice assures the
knowledge/technology transfer, the solutions feasibility and the personnel good will on University and
the implementation phase. Taken advantage of student’s irreverence, initiative, industrial SMEs
innovation spirit and work capacity, optimisation methodologies and systematic
studies can be developed. The technological and scientific strictness is assured by the collaboration
professor’s coordination. The companies have the opportunity to see and discuss the
diagnosis of their processes, to improve their process procedures, to have technological
and economical feasibility studies, among others, at low cost, in a short term, and with 57
results they can control. An additional benefit frequently observed is the contribution
of this type of projects to the initiation of a continuous improvement culture inside the
company.
The purposed model assures the response to the industry needs but also to the
university researcher’s need of papers publishing. The use of a scientific methodology
is a demand referred on the model. So, the university students and faculty benefit from
the interaction as they learn about the necessities and the improvement potential of the
industry and gain real-life exposure to practical problem-solving experiences that they
do not encounter in the classroom or in the laboratory.
The approach for collaborative projects between university and industrial companies
based on the proposed model requires best practices from both: companies must be
receptive to this way of know-how incorporation; universities must respect company
secrecy and fulfil strictly the project aims and delivery time.
The success of this type of projects strongly emphasise the findings of other
authors. The basic research is not always a necessary condition for innovation and
actually innovation can occur in complex interactions schemes between many factors
(Todd et al., 2001; Conceição et al., 1999). Demain (2001) refers that successful
interaction between academia and SMEs must rely on frequent contact in an
interpersonal based relation. Guthrie and Warda (2002) state that leadership is vital to
innovation and top management must be involved in the innovation process with
commitment and passion for change.
The conclusions withdraw from some comprehensive studies related with best
practices of collaboration between university and SMEs are also fulfilled by the
proposed model. Davenport et al. (1999) concluded that good will trust evolves and
produces positive results if repeated collaborative relationships occur. This result was
based on a study involving the development and implementation of a collaboration
procedure to facilitate communication between university and New Zeland’ SMEs.
Sheather et al. (1993) applied an experimental learning theory model for cooperative
education in Australia, with significant benefits for both industry and academia.
Cooperative education increased the Australian industry overall competitiveness in
global markets. Academia increased its knowledge and perception about the industrial
application of their own theoretical models and the students were trained under
industry-based scenarios, using real and industrial-useful case studies. Klofsten and
Jones-Evans (1996) based on a study developed under a ten-year period involving
European organizations, state that the collaboration between university and SMEs
must rely on five factors. The purposed activities/projects must meet real needs, be ran
by a core group and have a clear focus. The partners must have credibility and a close
relationship.
So, instead of huge and/or international projects building, the local collaboration
and partnership developed under this model can lead to continuously up-graded
BIJ companies and increased university societal framing (professors, researchers and
13,1/2 students). The collaboration between industrial companies and universities is a
foundation for competitive societies and, simultaneously, a tool for continuous
improvement of both organizations type (Lyu and Gunasekaran, 1993). Partnership
with a variety of manufacturers illustrates that university can provide industry with
practical knowledge and assist them within the processes of defining and
58 implementing state-of-the-art solutions for their current problems (Sheather et al.,
1993).

Model description
The following described model comprises three phases: preparation, diagnosis and
solutions development. In the preparation phase the university’s member (from now on
referred as professor) must establish contact with the industrial company top
management. The objectives of the project should be collaboratively defined after a
wide-ranging brainstorming concerning the company general requirements and
professor’s group expertises. It is critical for the project success to involve the company
top management at this phase, in order to assure both the employees active
involvement during the following phases and the matching between the project results
and company expectations. The project to be built must be localized, e.g. focusing a
workstation, a production cell, a small set of operations, etc. Trying to set up a complex
project in order to answer to several company expectations at the same time will result
in the project failure and/or in too long project duration, which is incompatible with the
students’ available time. If a company reveals expectations in several areas, then
several separated projects should be designed and performed independently.
Furthermore, if a company presents a complex and intricate problem it should be
decomposed in several subprojects, with interfaces well defined, but with objectives
and methodologies designed for a group of one or two students.
The professor can foreseen the work to be developed after company commitment
and interest in a project to tackle a specific problem. So he should run an informal
survey among the students in order to select an interested and psychological fitted one.
The student must be aware of industrial project typical features (e.g. several trips and
several time periods in the company, etc.), and also be prepared to deal with undercover
opposition by some of the employees. Also, the professor should select more than one
student (maximum three students) for projects needing higher levels of creativity
and/or work effort.
The next step is the definition of the project tasks and intermediate milestones. The
professor and the student(s), at the university side, and the head of product/production
sector (including elements of his team if he wishes so), at the company side, should be
involved in this step. The objectives of the defined tasks and the methodologies to
apply must be very clear, in order to simplify the communication and to avoid higher
levels of expectation than the ones withdrawal at the project end. This step is critical to
the professor: he must assure that it will be possible to develop the project using
structured, technical and up-to-date approaches and methodologies (e.g. Kanban,
SMED[1], JIT[2], TQM[3], TPM[4], MRP[5], discrete event simulation, etc.), within the
resources and time frame available.
The defined tasks and intermediate milestones should be submitted to the company
top management for approval. Remarks and/or inputs coming from the top
management should be taken into consideration, if the professor considers them University and
relevant and appropriate in a learning and research context. It should not be forgotten industrial SMEs
that the project content must not only aim the reduction or the elimination of the
company organisational and/or technical problems. It must allow the application of collaboration
universal scientific- and/or technical-based methods and tools and must promote their
incorporation as knowledge in the company.
After last step approval by both sides the project can start. The diagnosis phase 59
(phase 1 in Figure 1) begins with the information gathering from the manufacturing
system, work labour, product and process characteristics, etc. The students will lead
this phase by collecting data and asking for inputs from the production/product
responsible and from the technicians directly or indirectly involved.
Depending on the project type, these data can be qualitative and/or quantitative
(e.g. operations sequence, time study, layout interpretation, etc.). Preferentially the
information collected from the technicians and operators should be done at
the shop-floor and without the presence of the responsible (a formal meeting inhibits
the natural development of empathy between students and employees). This task of the
diagnosis phase must be compact: it should run for about two to three weeks, with at
least a four days-a-week presence of the students at the company to guarantee their
natural permanence in the manufacturing system environment. The success of this
task depends on the quality of the information gathered. So, students should transmit
to the company employees the interest of the project for all of them, demonstrating a
high level of motivation on running the different tasks. The professor must control
remotely this task, checking the relevance, the quality and quantity of the information
collected. Being aware of the student’s academy requirements (exams, classroom
attendance, laboratory work, etc.), the professor must coordinate the beginning of this
task in order to assure that it will be done without interruptions.
After collecting the information, the students can continue the diagnosis through the
identification of weaknesses, strengths, limitations, critical points, etc. and through the
purposing of general solutions – these two tasks should not be performed at company
premises. The outputs of this phase are a short and concise report (readable by the
company!) and, most important, an oral presentation of the first phase results in a
working meeting. Top management as well as the accompanying responsible and
technicians from the company and the professor must attend and actively participate in
this meeting. The main idea is to get involved all the intervenient of the decision-making
process. Using informal brainstorming methods the professor must assume the
leadership of the meeting. Beside collateral outputs (increase communication within
company hierarchy, promote people empowerment and continuous improvement) the
main meeting output is the identification of the solutions, understanding as action lines,
to be developed in the next phase.
The last phase (phase 2 in Figure 1) comprises the detailed development of the
solutions selected in the previous phase. Each solution must be described in detail and
its technical and economical impact estimated and analysed. So, in this phase are
developed either procedures or tools (mechanical systems, software, etc.), which must
be validated with the available data. Usually, there is a need for the students to spend
some more time in the company collecting new information or more precise data.
Depending on the solutions complexity these tasks can long about four to six weeks.
Nevertheless, generally, there is no formal need for regular visits to the company by
BIJ University Project Industrial company

13,1/2 Professor (s) Company Administration

general needs
vs.
university
expertise

60 Defination of project
objectives

Professor (s) Production/product

Analysis of the
Student (s) management team
needs,
constraints and
resources

Defination fo the project

tasks and milestones

Administration
Project approval
No
Yes Prodction/product
Student (s) Project start manager
technicians
Phase 1
• Gathering of the system/product ind=formation
• Diagnosis of the system/product
• Identification of critical points
• Purpose of general solution

Student (s)
1st Report presentation
Administration,
Professor (s)
management and
Student (s) Identification of technician
the solution to
Student (s) be developed

Phase 2
• Development of procedures, tools, software, etc.
• Validation of the developed system
• Technincal and economical impact analysis

Student (s)
2nd Report presentation

Administration,
Professor (s) management and
Selection of the
technician
solution to be
implemented

Project end

Implementation

Collaboration work with

Through company own
university and/or other
Figure 1. means
institutions
Model schematisation
the students. As in the former phase, the professor must remotely control the students’ University and
work and the validation criteria used. industrial SMEs
The presentation of the results of the solutions development phase is done using a
procedure similar to the one used in phase 1: written report and oral presentation for a collaboration
wide company audience. The impact of the developed proposals must be presented
under more than one scenario to allow the estimation of the potential implementation
benefits for the company in a simple but clear way. The aim is to provide the company 61
with systematised, simple and technically and/or scientifically supported information
to support the robustness of the decision-making process. Afterwards, company will
select some of the proposed solutions for future implementation, requesting the
professor opinion (or not). After the project the implementation phase can be performed
either through company own means or through a collaborative work with the
university and/or other institutions.

Case studies
This section presents several success case studies, where the described model and
collaborative work procedure were used. It is important to remark once again that
before the development of this model some unsuccessful cases occurred. The
unsuccessful was mainly related with the students psychological profile not assertive
enough to face challenges and overcome difficulties without loose a high motivation
level, with the inadequate expectations management carried out by the professor and,
finally, with the reduced commitment of company top management and collaborators
in the support of the project.

SMED case studies

Two projects based on the SMED[1] method were conducted at two individual
companies. The main aim was to increase the shop-floor productivity and flexibility.
The companies are very similar. Their main business is to supply automotive
assembly lines with sheet-metal forming parts.
Company A has about 350 employees and produces about 10,000 ton of parts per
year. Among other equipments this company has a total of 35 presses (mechanical and
hydraulic), with a nominal capacity between 20 and 800 ton. The project specific aims
were the development of a standardized procedure to simplify the tools design process
and also to reduce the presses setup time in 25 per cent without any investment
requirement (constraint imposed by the company). Two students in direct and close
relationship with the head of manufacturing sector of the company carried out the
project during ten months. It was developed an application tool (based on Microsoft
Excel) comprising, for all the press tools, geometrical and technological information
and standard best practices for the support of tool design process. The usefulness of
this software was high in what concerns the systematizing of design solutions and it
began to be used by the company immediately after its presentation and before the
project end.
For the SMED part of the project 19 presses were selected, particularly the ones with
a worst performance level and with a major productivity impact in the tool exchange
operation. Owing to similarities of some of the presses and/or of the setup
characteristics the 19 equipments were divided in three groups: smaller presses
(automatic feeding) with an average setup time of 57 minutes; larger presses (automatic
BIJ feeding) with an average setup time of 75 minutes; and manual feeding larger presses
13,1/2 with setup time around 30 minutes. At phase 1 the students conducted time
measurements as well as detailed identification of all operations performed during the
setup. A new setup procedure was developed at phase 2, for each press group, reducing
and/or eliminating internal operations and transforming some of the former internal
operation into external ones. The only costs involved were related with the personnel
62 training on the new procedures. As seen in Figure 2 the results of the project were
beyond the objectives, with setup time reductions between 50 and 60 per cent. Six
months after the end of project the company implemented these procedures and
extended them to the other presses.
Company B, although smaller, is similar to company A, working in the same
industrial sector and using the same core technologies. It has 140 employees and
produces around 7,500 ton of parts per year. Among other equipments this company
has a total of 45 presses (mechanical and hydraulic), with a nominal capacity between
10 and 800 ton. Also developed by two students during 12 months, this project aim was
to reduce setup to 50 per cent of the present time. From the preliminary observations
three groups of presses were identified: low capacity with automatic feeding, high
capacity with automatic feeding and low capacity with manual feeding. For a deep
analysis the four presses with more setup problems in each group were selected.
At the diagnosis phase, similar to the one performed in company A, the students
estimated setup time reductions under three scenarios for each press group. Scenario 1
only involves new work methods for the setup (modification of the operations and
resources organization); scenario 2 includes scenario 1 and requires the development
and/or acquisition of special systems to reduce internal operation time; including
previous scenarios, scenario 3 aims to reduce external operation through
transportation systems improvement, mainly between the tools warehouse and the
presses shop-floor.
The reduction in setup times achieved (between 68 and 86 per cent) was higher than
the one stated in the project objectives (Figure 3). For the phase of development the
company requested the students to develop the detailed setup procedure for all of the
press groups and also to perform a technical and economical analysis of the solutions
purposed under scenario 2 for the presses of the group 1. Company’s top management
decided to start the operators training on new setup procedures during the second
report presentation.

80 75.3
Before
70
After
60 57.0
–60%
Setup time (min)

50 –50%

40
30.6 29.4
30 28.0 –52%

Figure 2. 20 14.1
Setup time reduction in the 10
three press groups for
company A 0
Group 1 Group 2 Group 3
 120
–107.0 Before
University and
100 –77%
Scenario 1
Scenario 2 industrial SMEs
87.0
Setup time (min)
77.0
collaboration
80 –68%
–68%

60 –86% –77% –80% 63

40
24.3 27.5 24.3
20.0 Figure 3.
20 15.1 15.1 Setup time reduction in the
three press groups for
0 company B
Group 1 Group 2 Group 3

Lean assembly line case study

Company C among other activities like metalworking and industrial parts painting,
runs a production line in which medium and light-duty trucks are assembled, under a
components knock down approach. The truck’s parts are produced by a Japanese
manufacturer and arrive at company C as a “puzzle” to be assembled. The line is
composed by eight workstations where 16 operators perform assembly operations like
screwing, fixing, riveting and other essentially manual actions. The daily production
capacity before the project were launch was between four and six trucks per shift,
depending on the truck model.
The objective of this project was to increase the assembly-line productivity, in
particular to increase the line output without any increase in the resources, through the
implementation of lean manufacturing strategies. Since the company needed a rapid
answer to the problem, the work was performed during three months with the total
dedication of two students. At the diagnosis phase three main topics were analysed: the
layout, the materials flow for line-feed and the operation distribution and sequence at
each workstation. Using direct observation, work methods and time study (stop-watch
and work sample methods), the students identified critical operations, operation best
sequencing and best distribution within workstations. Through the application of
some lean manufacturing tools (5S, error-proofing, etc.) several simple and
ready-to-implement solutions were purposed (parts quantity-fitted containers,
coloured marks, one tool set per operator, etc.). With the information gathered about
factory plant dimensions and constraints and also about the materials transportation
volume between workstations, three alternative layouts were proposed.
After the first report presentation the company decided to implement the described
simple solutions and to proceed with the operations re-sequencing by their own means.
Therefore, the development phase involved the deep study of the impact of the three
layout alternatives in terms of total production time and cost. Applying a cell-based
philosophy to the workstation where the materials are pre-assembly and prepared to
feed the line and quantitative methods to analyse the workers and parts movements,
the three layouts were compared. Without giving deliberately one solution, the
comparison between the three was based on three strategic scenarios: investment-,
quality- and flexibility-based.
Owing to several constraints that did not exist at the time of the project, company C
implemented another layout, actually a mixed of the three presented. The overall
BIJ productivity of the assembly line increased about 45 per cent with a new production
13,1/2 rate not below eight trucks per shift.

Simulation case study

This project used software-based discrete event simulation in order to assess the
production system behaviour and the influence of some production parameters on its
64 productivity. Two students performed it during ten months in the spot welding section
of company A. This area receives the production orders on a daily base from the
production control manager (the orders are pulled by the clients since the company
works on a JIT[2] basis). Since there is a need to specific machine setup to each part
assembly (there are 120 different parts), an economic production batch, always higher
than the client order, is imposed. The extra quantities go to the section warehouse and
are considered WIP[6] assemblies (Plate 1). To perform the welded assemblies this
section receives un-welded parts from the press sections. The un-welded parts come in
batches sized by the presses economic production quantities. Therefore, at this section
warehouse there is a considerable amount of WIP parts, either before or after the
spot-welding operations.
The aim of the project was to define strategies to directly reduce the WIP in this
section and, indirectly, reduce the requirements of space for WIP storage and the total
production cycle time. The first task performed by the students was the information
gathering in order to describe, qualify and quantify the materials workflow. Following
they grouped the 120 different parts in 25 families allowing data to be processed in a
more rational way. Since several probabilistic variables were involved (order sequence,
order size, setup times for each machine and for each sequence, unexpected events in
the shop-floor), it was decided to use simulation as a sensitivity analysis tool. The
project key tasks were the information gathering from different sources, the algorithms
development and the manufacturing system modelling. This model was introduced in
a commercial software tool base on discrete event simulation and several analysis and
“what-if? . . . ” tests were done upon it (Figure 4).
The system behaviour was evaluated in the model in terms of the amount of WIP
parts and total production cycle time. Several sensitivity tests were run varying the
batches dimension, order sequence, order dimension and variability. Recommendations

Plate 1.
Parts container of WIP
parts just before the
spot-welding operations
 University and
industrial SMEs
collaboration

65

Figure 4.
Model interface image

were generated for each of the 25 assembly families. It was estimated that WIP of most of
the parts/assemblies could be reduced to between 80 and 40 per cent of the present. A
parallel result of the projects was a smooth introduction of a new analysis technique in
the company. The contact with discrete event simulation tools allowed a clear evaluation
of the benefits of such tools in an industrial environment where the productivity gains
are mainly achieved through the improvement of the materials flow.

Conclusions
The experience carried on by a research university group in the promotion of the
collaboration and teamwork attitude between the academia and SME companies,
involving researchers, young engineering students and SME employees and
top-management, was presented. The experience allowed the development of a
collaboration procedure where several phases and milestones were defined and some
best-practices were recommended. The collaboration towards a continuous
improvement and innovation process is promoted following a bottom-up approach.
The objective is to focus in localized and specific problematic areas in the industrial
companies where the potential of improvement and innovation is large, to diagnose the
situation and propose new and efficient solutions supported by technical/scientific
methodologies. Small projects allow both collaborators, academia SME companies, to
smoothly define their roles, achieve high levels of personal trust and design achievable
expectations within their competencies, which are the basic foundations to successfully
develop large and risky research projects.
The results achieved so far demonstrated a set of benefits:
.
training of young engineering students for an active problem-solving attitude,
within a systemic industrial perspective;
BIJ .
smoothness of the students transition to their professional life; and
13,1/2 .
promotion of a collaboration culture between SME and academia world for real
problems-solving and for continuous improvement and innovation processes.

Notes
66 1. SMED – single minute exchange of die.
2. JIT – just-in-time.
3. TQM – total quality management.
4. TPM – total productive maintenance.
5. MRP – manufacturing resource planning.
6. WIP – work in process.

References
Alazmi, M. and Zairi, M. (2003), “Knowledge management critical success factors”, Total Quality
Management & Business Excellence, Vol. 12 No. 2, pp. 199-204.
Cahill, E. (2001), “Breaking the mould: European manufacturing in the knowledge economy”,
Foresight, Vol. 3 No. 4, pp. 297-308.
Conceição, P., Heitor, M.V. and Oliveira, P. (1999), University-based Technology Licensing in the
Knowledge Based Economy, Techovation, Lisbon.
Davenport, S., Davies, J. and Grimes, C. (1999), “Collaborative research programmes: building
trust from difference”, Technovation, Vol. 19 No. 1, pp. 31-40.
Demain, A.L. (2001), “The relationship between universities and industry: the American
university perspective”, Food Technology and Biotechnology, Vol. 39 No. 3, pp. 157-60.
Graff, G., Heiman, A. and Ziberman, D. (2002), “University research and offices of technology
transfer”, California Management Review, Vol. 45 No. 1, pp. 88-115.
Guthrie, B. and Warda, J. (2002), “The road to global best: leadership, innovation and corporate
culture”, The Conference Board of Canada, Innovation Challenge May, Paper #1, pp. 1-8,
available at: www.conferenceboard.ca/inn/expertise/mng_tech.htm
Harmon, B., Ardishvili, A., Cardozo, R., Elder, T., Leuthold, J., Parshall, J., Raghian, M. and Smith,
D. (1997), “Mapping the university technology transfer process”, Journal of Business
Venturing, Vol. 12 No. 6, pp. 423-34.
Jones-Evans, D., Klofsten, M., Andersson, E. and Pandya, D. (1999), “Creating a bridge between
university and industry in small European countries: the role of the industrial liaison
office”, R&D Management, Vol. 29 No. 1, pp. 47-56.
Keeble, D., Lawson, C., Moore, B. and Wilkinson, F. (1999), “Collective learning processes,
networking and ‘institutional thickness’ in the Cambridge region”, Regional Studies, Vol. 33
No. 4, pp. 319-32.
Klofsten, M. and Jones-Evans, D. (1996), “Stimulation of technology-based small firms – a case
study of university-industry cooperation”, Technovation, Vol. 16 No. 4, pp. 187-93.
Liyanage, S. and Mitchell, H. (1994), “Strategic management of interactions at the
academic-industry interface”, Technovation, Vol. 14 No. 10, pp. 641-55.
Lyu, J. and Gunasekaran, A. (1993), “Implementation of advanced manufacturing technology
trough industry government university cooperation in Taiwan”, Computers in Industry,
Vol. 22 No. 2, pp. 187-91.
Mateus, A. (2000), Diagnóstico Prospectivo de Indústria Metalúrgica e Metalomecânica University and
Portuguesa, Associação Nacional das Empresas Metalúrgicas e Metalomecânicas, Lisboa.
Matlay, H. and Hyland, T. (1999), “Small firms and the university for industry: an appraisal”,
industrial SMEs
Educational Studies, Vol. 25 No. 3, pp. 253-67. collaboration
Mesquita, R., Peças, P. and Gomes, S. (2000), “Situação e Tendências do Sector Metalomecânico”,
in Tavares, L.V. (Ed.), A Engenharia e a Tecnologia ao Serviço do Desenvolvimento de
Portugal, Editorial Verbo, Lisboa, pp. 295-302. 67
Murphy, J. (1993), “A degree of waste – the economic-benefits of educational expansion”, Oxford
Review of Education, Vol. 19 No. 1, pp. 9-31.
National Research Council (NRC) (1998), Visionary Manufacturing Challenges for 2020, National
Academy of Science, Washington, DC.
Santoro, M.D. and Chakrabarti, A.K. (2001), “Corporate strategy objectives for establishing
relationships with university research centers”, IEE Transactions on Engineering
Management, Vol. 48 No. 2, pp. 157-63.
Sheather, G., Martin, R. and Harris, D. (1993), “Partners in excellence – a new model for
cooperative education”, Educational & Training Technology International, Vol. 30 No. 1,
pp. 5-31.
Tavares, L.V. (2000), A Engenharia e a Tecnologia ao Serviço do Desenvolvimento de Portugal,
Editorial Verbo, Lisbon.
Todd, R.H., Red, W.E., Magleby, S.P. and Coe, S. (2001), “Manufacturing: a strategic opportunity
for engineering education”, Journal of Engineering Education, Vol. 90 No. 3, pp. 397-405.

About the authors

P. Peças is a Research Assistant at Mechanical Department of Instituto Superior Técnico. Paulo
Miguel Nogueira Peças received the Master’s Thesis on Laser Welding of Taylor-Blanks (1998)
and also a PhD student (Thesis theme: Technology transfer to mould industry). Several papers
were published in international journals and international conferences. Paulo Miguel Nogueira
Peças is the corresponding author and can be contacted at: ppecas@ist.utl.pt
E. Henriques is an Assistant Professor at Mechanical Department of Instituto Superior
Técnico. The author completed the PhD thesis on Integrated Process and Production Planning
(2000). The main areas of interest are agile and robust manufacturing systems and technology
transfer to SMEs. Also, several papers were published in international journals and international
conferences.

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

BIJ
13,1/2 Global benchmarking for internet
and e-commerce applications
A.M. Ahmed
68 The European Centre for Total Quality Management,
University of Bradford School of Management, Bradford, UK
M. Zairi
Best Practice Management, e-TQM College, Dubai, UAE, and
S.A. Alwabel
The European Centre for Total Quality Management,
University of Bradford School of Management, Bradford, UK

Abstract
Purpose – To examine issues related to the development of the internet and e-commerce (EC) in the
Kingdom of Saudi Arabia.
Design/methodology/approach – Two surveys were sent out following a pilot project to gather
data on the validity and reliability of the questionnaires. The first survey was sent to 60 internet users,
a total of 48 responded with a rate of 80 per cent. The second was sent to 60 managers and accountants
within organisations in Saudi Arabia, a total of 44 responded with a rate of 73 per cent.
Findings – The key challenges identified for Saudi’s organisations are the continuing relying on
face-to-face contact principles, problems with information overload, charges still expensive, the need
for technical support and expertise, lack of a management commitment and understanding the
potential role of information technology (IT) on the country’s future and middle aged and older people
were more reluctant to use IT.
Research limitations/implications – Difficulties in reaching the right people to answer questions.
This paper was relying on the subjective opinions and generlisability issues and it could benefited
from further analysis.
Practical implications – For executives, the implications of the findings are that those factors
identified can be used as a checklist to assist companies in their effective adoption of e-business and
the maximisation of opportunities.
Originality/value – This paper is an initial phase of an on going research, which will contribute to
the body of knowledge in the EC domain from different cultural perspectives.
Keywords Benchmarking, Electronic commerce, Performance management, Best practice, Internet,
Saudi Arabia
Paper type Research paper

Benchmarking: An International
Journal
Vol. 13 No. 1/2, 2006
pp. 68-80
q Emerald Group Publishing Limited
1463-5771
DOI 10.1108/14635770610644583
Introduction
Internet together with other information and communication technologies are not
only driving financial global economies but also transforming societies into
knowledge-based economies around the world. Recognising the vast potential of
these technologies to better the lives of people, American, European and far east
governments are investing substantially on different initiatives to harness the full
power of the internet.
Over the last four years, the internet has shown an impressive growth record in
terms of commercial trade volume and private users (OECD, 2000). This has led to
redefining almost all aspects of businesses. For example, the adoption of recent and Global
advanced technologies in the way internet is functioning enabled organisations in benchmarking
facilitating transactions in a speedy manner, lowering the search costs and perfecting
the match between both buyers and sellers. Therefore, on one hand, internet is playing
a key role in transforming social patterns, strengthening the trade links between
nations as well as widening opportunities for economic and social development
worldwide. On the other hand and from the organisation’s perspective this has resulted 69
in a fierce price competition, diminishing product differentiation and disappearance of
brand loyalty (Brynjolfsson and Michael, 2000).
Bakos (1998) indicates that the use of internet opened many venues for buyers to
find almost every product available for trading online, which can be bought in common
markets. Carlson et al. (2001) points out that the existing consumer tracking technology
helps to generate useful data about consumer preferences and the resulting customer
profiles allows the creation of products, which are tailored to the buyers’ needs.
Matthew (1998) emphasises that internet has made new products available, which did
not and could not exist before. These products are known as digital products, as they
have no physical form in production and use, but primarily exist in the form of small
pieces.
The basic idea of benchmarking is: categorise and respond. Snowden has postulated
that known space is the domain of good practice. Within known limits we can both
predict and prescribe behaviour (Snowden, 2002). Benchmarking as a term should
motivate political decision makers because it a positive activity perceived as a
mechanism for improving operations and policies to proactively search for best
practices and management systems. Benchmarking is not a cookbook process that
requires only looking up ingredients and using them for success. Benchmarking should
be a discovery process and a learning experience. It requires observing what the best
practices are and projecting what performance should be in the future. Benchmarking
is a winning economic policy strategy, because it assists political leaders and policy
analysts in identifying practices that can be adapted to build winning, credible,
defensible plans and strategies, and complement new initiatives to achieve superior
performance (Ahokas and Kaivo-Oja, 2003).
Moreover, benchmarking, according to McGaughey (2002), is an important
methodology for establishing suitable performance standards to guide companies in
developing strategies and systems for the successful implementation of e-commerce
(EC). This paper will mainly focus on what the concept of EC actually involves. This
will be clarified by firstly explain the factors that contribute to internet growth.
Moreover, internet key applications will be examined to get a clear overview of what it
is all about. Several secondary case studies regarding internet, from over the world, are
revised in order to get a clear picture of people perspective of the internet. By analysing
these case studies, it can be easy to understand the opportunities and threats of
e-business and how industries can utilise these benefits and overcome these barriers.

Factors contributing to internet growth

According to Veloso (2000) there are three factors contributing to the growth of internet
commerce. First is the continuous decline in the prices of information technology (IT)
products, such as computers and software’s. Second, the development of different
platforms and mass distribution of internet browsers such as Netscape, which provide
BIJ a relatively easy way to firms to develop a user-friendly interfaces (web sites). Third
13,1/2 factor is the commercialisation of the web itself with media-rich content and EC.
Figure 1 shows the distribution of these users by region and the statistics shows
that the majority of internet users are concentrated in the USA, but there is also
evidence that Japan and Western Europe are catching up fast. Forecasters predict that
the more accelerated growth over the next three to five years is expected to take place
70 in Asia and Latin America.

Internet key applications

EC and other internet applications will never reach their expected potential unless
consumers have confidence and security in going online. EC is a shorthand term that
embraces a complex mixture of technologies, infrastructures, processes, and products
brings together whole industries and narrow applications, producers and users,
information exchange and economic activity into a global marketplace called “the
Internet”. There is no universal definition of EC because the internet as a marketplace
and its participants are frequent and their intricate relationships are evolving rapidly.
Rous (1999) states that EC is an internet application, running on an infrastructure
composed of computers, communication systems and software, it uses the internet’s
most important infrastructure applications like the browser, the world wide web and
e-mail. It enables companies and individuals to break down geographical limits and
time differences combined with the relatively low cost of accessibility. It is estimated
that revenue form using EC will reach $1 trillion in 2003-2005. Therefore, Odlyzko
(2000) predicted that traffic on the internet is estimated to double each year.

Business-to-business (B2B)
EC is an umbrella term that covers virtually all forms of electronic trading (Greek,
1998). The essence of EC is buying, selling and marketing on the internet (House of
Lords Select Committee on the European Union, 2000). EC is a subset of e-business.
A company is engaging in e-business to the extent that EC pervades fundamental
business processes and activities. B2B EC is the network supported buying, selling,
marketing and supporting of products (goods and services) by businesses. Electronic
networks such as intranets, extranets and the internet support communications and
transactions among trading partners. The infrastructure resulting from various
combinations of the above-mentioned networks facilitates the exchange of data,
information, money, and in some cases products, within and among organisations.
B2B is an activity that, from the standpoint of most final consumers, takes
place behind the scenes. Most final consumers care little about the supply chains

6% Canada and USA

23% South Africa
41%
Africa

Figure 1. Asia-Pacific
Internet users by region Europe
percentage 23% 3% Middel East
6%
that bring them products, or the role of B2B in those supply chains. The primary Global
concern of final consumers is value, getting the best quality for the lowest possible benchmarking
price. Although final consumers probably do not know or care that B2B can have
a very significant impact on the value of products they purchase, many businesses
do! Manufacturers, retailers and service providers now recognize the potential of
B2B for improving supply chain performance and ultimately the value delivered to
final consumers. This may well be the reason that B2B is the fasting growing area 71
of EC.
There are six subcategories of e-business. They are as follows: B2B;
business-to-consumer (B2C); government-to-consumer (G2C); government-to-business
(G2B); government-to-government (G2G); and consumer-to-consumer (C2C).
Big businesses first used B2B extensively to buy and sell industrial products, but
B2B is no longer restricted to large companies (Mayer-Guell, 2001). A recent Dunn and
Bradstreet survey in 2001 of small businesses in the USA, found that one-fourth of the
survey respondents were engaged in B2B. Recent developments have accelerated the
trend towards B2B. The advent of centralized internet sites called exchanges, that
allow small businesses to pool their buying power, is contributing to the spread of B2B
to small businesses (Mayer-Guell, 2001).
Gartner Group (2000) has forecast that by one year later, 2004, the global B2B EC
market will have continued growing and will be worth £4.8 trillion. As James (2000)
has remarked, it is hard to know how seriously to take these dramatic predictions but
major growth in this area seems inevitable.
The first major developments in the area of buying and selling on the net (B2B)
started in the mid-1990s and saw major firms such as Wal-Mart and General Electric
(GE) moving buying and selling online to cut costs and speed supplies. The aims of
cutting paperwork and time may have been simple but the results were impressive. GE
demonstrates another example. The company has a trading process network, which is
a web-based link to suppliers so that they can bid for GE components’ contracts.
According to GE, it cut procurement cycles in half, processing costs by one-third and
the cost of goods purchased by between 5 and 50 per cent.
Research has shown that organisations often develop a presence on the net without
paying sufficient attention to the quality of their web site. The have clearly outlined the
importance of a number of factors including clarity of purpose, accessibility and speed
and ease of use as well as site content and the way in which the site is related to
customer service and relationships. Too often, those operating in the B2B market think
their web site design is relatively unimportant compared to more consumer-oriented
sites (Cox and Dale, 2001; Timmers, 1999; Greenspan, 2000; Varian, 1997;
Wyckoff, 1997

Business-to-consumer (B2C)
EC began as a vehicle for browsing the web and buying a few books or CDs. This has
been transformed into a platform that changed how organisations and customers are
interacting virtually. In its early stage, EC was adopted to support B2C information
flows and simple transactions. B2C EC now enables consumers to read and listen to
national and international news, purchase products, make investments and apply for
loans as well as engaging in countless other information- and entertainment-related
activities.
BIJ There are a number of economic forces that affect B2C EC. Levels of discretionary
13,1/2 income, credit card use, telephone access, personal computer ownership, and internet
service subscriptions are important indicators, and tend to create a gap between
various tiers of living standard. More affluent citizens, for example, tend to live in cities
rather than in rural areas. Other forces are discussed in different cases in the following
sections.
72
The Brazil and Singapore cases
The majority of web users in Brazil are from the upper income classes because they
speak English and are wealthy. However, the case in Singapore is different. A recent
survey conducted by Lien (2000) estimated that 59 per cent of the country’s
households own PCs and 42 per cent have internet access. But this high level of PC
ownership does not necessarily translate into equally large percentages of consumer
EC. Even though the Singapore rate exceeds such percentages for Japan, Australia,
and the USA, consumer EC in Singapore is still in its early stage. High internet
access costs and low service quality might discourage widespread adoption of
consumer EC.
There appear to be two significant aspects of culture that impact B2C EC:
(1) consumer trust in the accuracy and privacy of the electronic transaction; and
(2) a preference for face-to-face purchasing transactions (China Online, 2000; Katz
and Malkin, 1999; Colitt, 1999).

A Goldman Sachs Investment Research report claims that the five-year (1998-2003)
growth rate of internet use in Asia is expected to be twice to USA (Asia Pulse, 1999). In
both absolute numbers and percentages, the greatest potential for B2C EC rest in Asia.
There are four key variables of Asia culture can be identified from these projections.
First, Asians generally are not always comfortable with the idea of being in debts so
few people have credit cards. Yet the primary means of payment for B2C EC is a credit
card, thus potentially eliminating a large segment of the population from engaging in
EC. This is particularly true in China (Inside China Business News, 1999; Stirpe, 1999),
a country that is forecast by IDC to have 18 per cent of the Asia-Pacific region’s
internet users in 2003, and to account for US$11.7 billion in online transactions in 2004
(Singapore Business Times, 2000).
Second, Asians are less trusting and inclined to share personal information with
retailers, especially in online transactions without face-to-face contact (Flagg, 1999;
China Online, 2000). Third, Asians ability to provide a reliable and affordable
telecommunications infrastructure, and its physical transportation and delivery
systems are not yet completed. While the projection of South Korean internet use in
2003 (China Online, 2000) is small in number, approximately 9.5 million users, a strong
interest in wireless phone subscriptions and the growth of public telecommunications
networks, domestic online services, and internet access will help build South Korean
consumers’ use of EC (Seo, 1999). Singapore recently loosened its regulations on
the telecommunications industry, potentially creating a stronger and more
competitive market for telecommunications and internet services (Borsuk, 2000).
The availability of free internet access and e-mail accounts, along with more reliable
and affordable fixed and mobile phone systems, might make consumer EC more
attractive to Asians.
 Fourth, the uncertainty and risk related to Asia’s business and economic climate. Global
The market economies of Japan, South Korea, and Singapore are recovering from the benchmarking
Asian recession of the late 1990s. These areas seem ripe for the development of B2C
EC, as consumer interest in the internet grows. Singapore’s government seeks to
promote a broader, contemporary telecommunications infrastructure and to promote
internet business development. While the Chinese economy was less affected by the
1997 financial crisis, the volatility of its government policies and business initiatives 73
has caused some firms to exercise caution and move slowly with their EC plans (CNN
Financial News Online, 1999; Flagg, 1999; CNNIC, 2000).

Europe case
According to the Pan European Internet Monitor, 107.8 million Europeans, or 34 per
cent of the total population, have internet access (Nua web site, 2000). This number is
expected to grow to 50 per cent by 2003. Jupiter Communications, an EC research firm,
projects that EC revenues in Europe are set to increase eightfold, rising from
approximately EUR 8 billion in 2000 to EUR 64 billion by 2005. However, a lack of
initiative dealing with consumers’ privacy questions has the potential to negatively
impact both European internet commerce and online advertising. For example, only
10 per cent of European web sites surveyed by Jupiter Communications (2000) post a
privacy policy on or linked from the main homepage.
Liesbeth Hop, CEO of Pro Active International, stated:
These first results of the Pan European Internet Monitor show that Internet in Europe is
starting to catch up at high speed, although we still see very large differences between
Northern, Southern and Western Europe (Nua web site, 2000).
A total of 35 per cent of adults in the UK and 30 per cent of adults in Germany have access
to the internet. Contrast that with the 5 per cent of adults in Hungary who have internet
access, and the digital divides becomes very apparent in Europe (CyberAtlas Web site,
1999). Germany is projected to be Europe’s largest EC marketplace (Andersen, 1999).

The Latin American case

In one respect, the opportunity for B2C EC in Latin America is similar to that in Asia: a
large percentage of the population is not yet connected to the internet. Some are
optimistic about Latin America’s high growth potential for computer hardware
companies and internet service providers. IDC reports that the number of internet users
will reach 19 million by 2003, compared with 7.5 million users in 1999. Jack Davies,
president of America Online International, says, “The market is less well developed
than Europe, Japan, or Canada, but there’s a bigger upside potential” (Katz and Malkin,
1999). Latin America’s demographics also speak to the promising future of EC, says
Susan Segal, General Partner and Head of Latin America at Chase Capital Partners.
Nearly about 100 million people control 65 per cent of the region’s buying power, and
75 per cent of Latin Americans fall in the internet use age category of 15-34-year-olds
(Latin Finance, 1999).

The North America and Canada cases

Ninety-two million people in the USA and Canada, or 40 per cent of the population
over the age of 16, report that they use the web (Thompson, 1999), and the USA alone
BIJ was expected to have 115 million internet users by the end of 1999 (CyberAtlas
13,1/2 Web site, 1999).
The only regularly predictable variable in North American e-business is change.
Internet use in the USA and Canada has continuously grown and evolved from its early
public adoption. While Canada is not far behind, the top ten web sites visited (by
volume of unique hits) are still all based in the USA, supporting the idea that the USA
74 is a hub of e-access and commerce.
In Canada, this aspect of EC is different from its southern neighbour. In Canada’s
vision for the future of electronic information services, a “government as a user” simile
is used to project its view of uses and applications for information exchange (Canadian
Information Highway, 1997).
The USA and Canadian cultures further contribute to a successful e-business
paradigm because debt is an accepted part of their consumer culture. Singapore’s
consumers have successfully adopted a “card culture”, using debit rather than
credit cards (Northern Light, 1995). Still, the USA and Canada have been slow to shift
to a debit card culture. Such a fondness for the credit card has been taken as far as a
consumer’s interaction with the federal government. Recently, the US
government began accepting credit cards as a means of paying federal income taxes
(Kadlec, 1999).

Electronic finance (e-finance)

In the last 20 years, the financial services industry has witnessed dramatic changes,
largely driven by globalisation, deregulation and consolidation trends. Technological
advances accelerated the process, reinforcing some of these trends particularly
globalisation and deregulation, as well as facilitating the development of e-finance
(Nieto, 2001; Saatcioglu et al., 2001). This has blurred the boundaries between different
financial institutions, enabling new financial products and services to appear and
making the existing one available in different packages.

Electronic banking (e-banking)

E-banking refers to the use of internet as a remote delivery channel for providing
services such as opening a deposit account, transferring funds among different
accounts and electronic bill presentment and payment. This can be offered in two
main ways. First, an existing bank with physical offices can establish a web site
and offer these services to its customers in addition to its traditional delivery
channels. Second, is to establish a virtual bank, where the computer server is
housed in an office that serves as the legal address of such a bank. Virtual banks
offer their customers the ability to make deposits and withdraw funds via
automated teller machines or other remote delivery channels owned by other
institutions.
Banks in Saudi Arabia are moving slowly and with caution into the e-banking
because their concerns for security and reliability. There are a multiple calls for
academicians, computer specialists, and people at large to pay serious attention to
banking system and increase awareness of its implication on the country future. Once
customers are convinced about the multifarious advantages of e-banking they will
move ahead to the virtual bank system.
Research methodology Global
The starting point was the explorative approach. Following literature review the benchmarking
research approach has changed into an explanatory descriptive. The e-mail
questionnaire was developed based on the Karahanna et al.’s (1999) questionnaire
and follows the scales guidelines suggested by Ajzen and Fishbein (1980) model. Thus,
most of the items in the questionnaire was validated and checked for reliability. All the
questions were structured using the five-point Likert scales, which numerically grades 75
responses, with 5 representing “Strongly agree”.

Data collection
A pilot sample was sent to academic within the university as an initial step to ensure
the accuracy and relevance of questions. After reviewing the pilot, it was observed that
the procedure matches what has been suggested by Saunders et al. (2000) when using
similar administrative questionnaire via e-mail. Therefore, prior sending the
questionnaire to managers and accountants within organisations in Saudi Arabia
and to ensure better response rate, the authors made contacts with those organisations
to encourage and provide awareness of the research values. These organisations
environments had a number of networked computers, with internet resources, that
were used substantially in organising their programmes.
The second step was sending the questionnaire via e-mail, a sample of 60 internet
users and 60 managers and accountants accompanied by a covering letter. The third
step, e-mailing those responded first and thanking them for the prompt reply. The
fourth step was sending a follow-up sample with a covering letter and copy of the
questionnaire to those who have not responded.
The survey instruments asked for three types of information relating to
organisations and internet users in Saudi Arabia as they relate to Gardiner’s (1989)
conceptual model and to the methodology used in researching them. These three types
of information concerning are: the use of IT (internet and EC), relevant psychosocial
factors in using the internet, and relevant physical factors in using the internet.
However, before going in depth in this information it is useful to be acquainted with
general information about the organisations and internet users that being surveyed.
This information includes whether the organisations have operated EC and how long
as well as information about the internet users’ gender, age, and educational level.

Empirical analysis
As shown in Figure 2, 45 per cent of respondents have indicated that their
organisations are using EC while 34 per cent did not. The remaining are planning to
use it in the near future.
About 25 per cent of organisations that have EC operated in their transactions
responded that, EC have been operating in their companies for less than six months,
while almost double this percentage said, they have it between year and two years, this
is shown in Figure 3. It is obvious from the chart that the percentage of operating EC
raises.
According to internet users’ survey, the greatest proportion of respondents as
illustrated in Table I was male (83.3 per cent) this due to the cultural of Saudi people.
Figure 4 shows the age of the respondents in six-year increments, starting with the
category of below 18 years. Internet users aged 31-45 years represent the largest
BIJ 45%
13,1/2
40%
35%
30%
76 25%
20%
15%
10%
5%
0%
Figure 2.
Organisations operating Yes
EC No
Will be

15% 15%
More than two years
A year to two years
Less than 6 months
Figure 3. 25%
Period of operating EC 7 months to a year
45%

Gender Number Per cent

Table I. Male 40 83.3

Gender of respondents Female 8 16.7

45.00%
40.00%
35.00%
30.00%
25.00%
20.00%
15.00%
10.00%
5.00%
0.00%
Under
19 to 30
18 31 to 45
Figure 4. 46 to 60
Internet users age group above
60
category and the smallest were under the age of 18 years. This can be due to the Global
awareness of the benefit of the internet in that age. benchmarking
Figure 5 shows education level of respondents. The majority of internet users in this
sample hold baccalaureate (33.33 per cent), and (27.08 per cent) of the respondents have
a high school qualification.
Figure 6 shows the length of respondents being using internet to reflect their
experience and difficulties are facing. About 53 per cent of the users only being using 77
the internet less than six months and only 23 per cent have been using it more than
three years. This indicates that internet usage is still in a very early stage of
development.

Discussion and conclusions

To examine the potential of EC, business must be aware of the benefits and challenges
of trading electronically. This research examined these factors in the Saudi’s
companies. The key challenges identified for these organisations are the continuing
relying on face-to-face contact principles, problems with information overload, charges
still expensive, the need for technical support and expertise, lack of a management
commitment and understanding the potential role of IT on the country’s future and
middle aged and older people were more reluctant to use IT.
The survey results confirm that the identified factors are similar across
organisations. For executives, the implications of the findings are that those factors
identified can be used as a checklist to assist companies in their effective adoption of
e-business and the maximisation of opportunities.

4.16% 6.25% High school

4.16% 27.08%
college 2 years
8.33%
Baccalaureate
Master
Doctoral PhD Figure 5.
Professional Internet users education
16.66% other
level
33.33%

6%
6%

10%
Less than six months
Seven months to a year
year to two years
53%
Two years to three years Figure 6.
More than three years The period of using the
23% internet
BIJ References
13,1/2 Ahokas, I. and Kaivo-Oja, J. (2003), “Benchmarking European Information Society
developments”, Foresight, Vol. 5 No. 1, pp. 44-54.
Ajzen, I. and Fishbein, M. (1980), Understanding Attitudes and Predicting Social Behavior,
Prentice-Hall, Englewood Cliffs, NJ.
Andersen, A. (1999), “Geographic differences in internet access among corporate executives
78 narrows”, Andersen Consulting News, available at: www.ac.com/news/5.99/news_051199.
html
Asia Pulse (1999), “Asian internet users to surge to 64 million by 2003: report”, available at:
http://etextb.ohiolnk.edu/
Bakos, Y. (1998), “Towards friction-free markets: the emerging role of electronic marketplaces on
the internet”, Communications of the ACM, Vol. 41 No. 8, pp. 35-42.
Borsuk, R. (2000), “Singapore opens telecommunications sector, drawing praise”, The Asian Wall
Street Journal, January, p. 7.
Brynjolfsson, E. and Michael, D. (2000), “Frictionless commerce? A comparison of internet and
conventional retailers”, MIT Sloan School of Management, Vol. 46 No. 4, pp. 563-85.
Canadian Information Highway (1997), “Preparing Canada for a digital world”, Canadian
Information Highway Advisory Council web site, available at: http://strategis.ic.gc.ca/ssg/
ih01650e.html
Carlson, J., Furst, K., William, W. and Daniel, E. (2001), “Internet banking: market
developments”, Proceedings of the Regulatory and Issues, Society of Government
Economists, United States.
China Online (2000), “China internet users hesitate to shop online”, available at: www.
chinaonline.com/topstories/000329/1/b1000032228.asp
CNN Financial News Online (1999), “China to regulate net boom”, CNN Financial News Online,
available at: http://gw.cnnfn.com/output/pfv/1999/12/13/emerging_markets/wires/
china_net_wg
CNNIC (2000), Semi-Annual Survey Report on Internet Development in China, CNNIC (China
Internet Network Information Center), available at: www.cnnic.net.cn/develst_e/
cnnic2000_e.htm
Colitt, R. (1999), “Consumers like their windows in the shops”, Financial Times, December, p. 13.
Cox, J. and Dale, B.G. (2001), “Service quality and e-commerce: an exploratory analysis”,
Managing Service Quality, Vol. 11 No. 2, pp. 121-31.
CyberAtlas Web site (1999), “US leads way in internet users”, available at: http://cyberatlas.
internet.com/big_picture/geographics/print/0,1323,5911_234841,00.html
Flagg, M. (1999), “Curtain rises on e-commerce”, The Asian Wall Street Journal, 1-7 November,
p. 1.
Gardiner, W.L. (1989), “Forecasting, planning and the future of information society”, in Goumain,
P. (Ed.), High Technology Workplace: Integrating Technology, Management, and Design
for Productive Work Environments, Van Nostrand Reinhold, New York, NY.
Gartner Group (2000), “Gartner group forecasts regional B2B outlook through 2004”, available at:
www.ncr.com/repository/research_reports/data_warehousing/gartner_dbcore.htm
Greek, D. (1998), “The road to net profit”, Professional Engineering, Vol. 11 No. 22, pp. 38-9.
Greenspan, A. (2000), “Global economic integration: opportunities and challenges”, available at:
www.federalreserve.gov.aug25
House of Lords Select Committee on the European Union (2000), E-commerce: Policy Development Global
and Co-ordination in the EU, Stationery Office, London.
benchmarking
Inside China Business News (1999), “Flurry of web sites thrusts China e-trade forward”, available
at: http://invest.insidechina.com/business.php3?id ¼ 107256
James, G. (2000), “B2B. Hot or hype?”, Supplement to Business 2.0, August.
Jupiter Communications (2000), available at: www.jup.com/jupiter/research/note.
jsp?doc ¼ mmsc99-01j 79
Kadlec, D. (1999), “IRS takes charge”, Time Online, available at: www.time.com/time/personal/
19990301/money.html
Karahanna, E., Straub, D.W. and Chervany, N.L. (1999), “Information technology adoption across
time: a cross-sectional comparison of pre-adoption and post-adoption beliefs”, MIS
Quarterly; Management Information Systems, Vol. 23 No. 2, pp. 183-213.
Katz, I. and Malkin, E. (1999), “Battle for the Latin American net”, Business Week, available at:
http://proquest.umi.com/
Latin Finance (1999), “A force for change”, Latin Finance Technological Edge Supplement,
September, Business & Industry database, available at: http://etextb.ohiolink.edu/
Lien, J. (2000), “E-commerce offered in only 5% of local companies’ web sites”, Singapore
Business Times, available at: http://business-times.asia1.com.sg/5/nsing/nsing04.html
McGaughey, R.E. (2002), “Benchmarking business-to-business electronic commerce”,
Benchmarking: An International Journal, Vol. 9 No. 5, pp. 474-84.
Matthew, W. (1998), “Digital delivery of cultural products on the electronic highway: the main
issues at stake with respect to the on-line delivery of cultural products”, Policy Options
-Montreal, Vol. 19 No. 5, pp. 30-2.
Mayer-Guell, A. (2001), “Business to business electronic commerce”, Management
Communication Quarterly, Vol. 14 No. 4, pp. 644-52.
Nieto, J. (2001), “Reflections on the regulatory approach to e-finance, bank for international
settlements”, CH-4002, Information, Press & Library Services, Basel.
Northern Light (1995), “Cards international”, available at: http://library.northernlight.com/
ck19980701060079423.html?cb ¼ 0&sc ¼ 0#doc
Nua web site (2000), “Proactive: digital divide still apparent in Europe”, available at: www.nua.ie/
surveys/index.cgi?f ¼ vs&art_id ¼ 905355714&rel ¼ true
Odlyzko, A. (2000), “The current state and likely evolution of the internet”, Proceedings of
Globecom’99, IEEE, 1869-1875.
OECD (2000), “Organisation for economic co-operation and development”, The Economic and
Social Impact of Electronic Commerce Preliminary Findings and Research Agenda.
Rous, B. (1999), “How to succeed in online markets. ACM: a case study”, The Journal of Electronic
Publishing, Vol. 4 No. 4, p. 10.
Saatcioglu, K., Stallaert, J. and Whinston, A. (2001), “Design of a financial portal”,
Communications of the ACM, Vol. 44 No. 6, pp. 33-8.
Saunders, M.N.K., Lewis, P. and Thornhill, A. (2000), Research Methods for Business Students,
Financial Times/Prentice-Hall, Harlow/Englewood Cliffs, NJ.
Seo, B. (1999), “Korean turnaround rings up telecom growth”, Electronic Engineering Times,
available at: http://etextb.ohiolink.edu/
Singapore Business Times (2000), “Top ten trends in IT industry”, Singapore Business Times,
available at: http://etextb.ohiolink.edu/
BIJ Snowden, D. (2002), “Complex acts of knowing: paradox and descriptive self-awareness”, Journal
of Knowledge Management, Vol. 6 No. 2, pp. 1-14.
13,1/2 Stirpe, A. (1999), “Chinese bombard the e-commerce market”, Computer Retail Week, available at:
http://etextb.ohiolink.edu/
Thompson, M. (1999), “The web: 92 million surfers strong”, IDG web site, available at: www.idg.
net/go.cgi?id ¼ 190770
80 Timmers, P. (1999), “Electronic commerce: strategies and models for business-to-business
trading”, Wiley, Chichester.
Varian, H.R. (1997), “Differential pricing and efficiency”, First Monday, No. 2, available at: www.
firstmonday.dk
Veloso, E. (2000), “The business revolution through B2B market tone and its impacts over the
financial system going into 21st century”, The George Washington University, School of
Business and Public Management, Washington, DC, available at: www.gwu.edu/ , ibi/
minerva/fall2000/eveline.isidoro.pdf
Wyckoff, A. (1997), “Imagine the impact of electronic commerce”, OECD Observer, Vol. 208.

Further reading
Eid, R., Trueman, M. and Ahmed, A.M. (2002), “A cross-industry review of B2B critical success
factors”, Internet Research: Electronic Networking Applications and Policy, Vol. 12 No. 2,
pp. 110-23.
Forrester Research (2000), “E-marketplaces will lead US business e-commerce to $2.7 trillion in
2004”, available at: www.forrester.com/er/press/release/0,1769,243,ff.html

Corresponding author
A.M. Ahmed can be contacted at: a.m.ahmed1@bradford.ac.uk

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

E-commerce
Value decomposition of performance
e-commerce performance
Dieter Fink
School of MIS, Edith Cowan University, Churchlands, Australia 81
Abstract
Purpose – The need to measure organisational effectiveness has become increasingly accepted.
Performance measurements can readily be developed but need to be further defined as performance
metrics. The paper’s aim is to show how the use of the decomposition technique enables the
development of electronic commerce (e-commerce) metrics.
Design/methodology/approach – In the approach to value decomposition, value constructs are
identified at the highest level, decomposed into the next levels, referred to as value variables, which are
further broken down into value metrics. During decomposition, proper splicing has to occur. This
implies that values (constructs, variables and metrics) should be carefully specified and that their
interconnections should be controlled. Furthermore, functional integrity is required so that a succinct
statement can be made for each metric.
Findings – The paper identified value constructs and value variables for six e-commerce
applications. They are aggregated into corporate performance management measures so that the
various lower level value metrics can be made to work seamlessly together.
Practical implications – Although there is large agreement on the key values obtained from
e-commerce, it is difficult to achieve the degree of integrity required by the decomposition technique.
Further research is required.
Originality/value – Performance measures provide a factual representation of important business
activities and enable greater preciseness in their execution. The paper provides insight into values and
associated measures in important activities of e-commerce, namely VRM, B2C, B2B, CRM, EP, and
ERP.
Keywords Electronic commerce, Performance measures, Organizational effectiveness
Paper type Research paper

Introduction
With older forms of information technology (IT), the returns flowing from the
investment were referred to as benefits and included in a cost benefit analysis (CBA).
Benefits could relatively easily be quantified in monetary terms since IT produced
tangible benefits mainly in the form of cost reduction (e.g. the computer which
produced billings replaced x number of billing clerks) and better utilisation of working
capital (e.g. reduction in inventory through automated inventory management).
With the emerging new economy, the simplistic accounting-oriented CBA approach
no longer suffices since there are new ways of doing business where benefits are
closely intertwined with business activities:
New web technologies are offering companies unprecedented opportunities to rethink
strategic business models, processes and relationships (Feeney, 2001, p. 41). Benchmarking: An International
Journal
Vol. 13 No. 1/2, 2006
Furthermore, rather than measuring dollar values as did CBA, metrics operationalise pp. 81-92
electronic commerce (e-commerce) performances that are non-monetary in nature but q Emerald Group Publishing Limited
1463-5771
which nevertheless contribute to the overall profitability of the organisation: DOI 10.1108/14635770610644592
BIJ Simply put, if you can develop and apply reasonable metrics to intangibles such as customer
satisfaction, communication, decision cycle time, decision-making quality, cycle time,
13,1/2 delivery performance, etc. then it is very likely the economic impact can be reasonably
predicted (Donovan, n.d.).

82 Performance measurement
Business people constantly talk about organisational performance; hence the need to
measure organisational effectiveness has become increasingly accepted. As indicated
above, performance measurements can be developed in broad areas of business
activity, such as providing customer service, but need to be further defined as
performance metrics. The latter are:
. . . standards of measurements, such as the maximum acceptable elapsed time to complete a
transaction, used to establish a benchmark target that is compared to actual performance
(Gouscos et al., 2003, p. 51).
Metrics provide a factual representation of important business activities and enable
greater preciseness in their execution. Metrics can be used in two ways (Weischedel
et al., 2003). First, they are used to assess internal performances. In this way they
provide feedback on achievements, for example, to determine performance bonuses,
and can be the basis for strategies and actions, for example, to introduce business
change. Second, they can be used as benchmarks from which comparisons are made
with industry and other norms. The emergence of metrics has furthermore improved
the perception of greater corporate transparency.
Tried and proven approaches in the form of traditional accounting metrics (e.g.
return on investment, net present value, payback) to justifying and measuring the
payoffs of investments currently exist in organisations. They have become a
convenient and familiar approach to most business executives but, because of their
financial nature, they tend to have a short-term focus. They do not readily have the
ability to measure non-financial business values which tend to have a long-term focus.
This has caused a widening gap between the real, market-determined value of the
organisation and its book, accounting-derived value. The Gartner Group (Koulbanis,
2003) estimates market value to be three times the book value as of July 2003 for S7P
500 organisations. What is required are metrics that measure true value; however, few
firms so far have developed adequate performance metrics related to IT value
(Weischedel, 2003).

The technique of decomposition

The technique of decomposition is well known in the information systems and
computer science disciplines. It is commonly applied during the phases of systems
analysis in producing data flow diagrams in which a system is broken down into
smaller and smaller pieces (Eliason, 1987). During systems design it is practised in the
normalisation of data to reduce redundancy (Hawryszkiewycz, 1994) and in the
construction of structure charts to reduce complexity in the cohesion and coupling of
the data (Davis, 1994). Essentially, the technique of decomposition is a top-down
approach to solving a complex problem. Gillettt and Pollack (1982, p. 25) provide an
excellent outline of the concept as follows:
 The basic concept in the top-down design process is to take the original problem and perform E-commerce
a stepwise refinement in which the problem is segmented or decomposed into smaller,
functionally self-contained subproblems, each of which is simpler to solve than the original performance
one. This must be done in such a way that when these component subproblems are “spliced”
together, they correctly solve the original problem. Proper splicing of component
subproblems requires careful specification of the data and control interfaces between them.
In the decomposition process, the concept of functional integrity should be stressed.
Associated with each subcomponent, it should be possible to make a succinct statement about 83
the function it performs or the problem it solves: the composition does “X”. This function or
problem solution should make sense to an interested observer and should embody a complete
self-contained activity. Such decomposition makes it easy to splice the components back
together and verify that the internal algorithm chosen to carry out the function is correct.
Once the stepwise refinement has been performed at a specific level (on a specific problem),
each of the component subproblems can be addressed individually as a problem in its own
right. This the process can repeatedly be applied to each of the subproblems until a desired
level of decomposition is reached.
The above approach appears intuitively well suited to solving the problem of
developing performance measures for e-commerce. To begin with, the broad values of
an e-commerce application are identified. This is relatively easily done because they
are widely published and disseminated. However, because they are broadly stated they
are not meaningful for the purpose of developing a performance justification. For
example, the promise of business-to-consumer (B2C) e-commerce to provide better
customer service is only realisable if the concept is broken down into activities that
deliver customer service.
To produce a meaningful measure of performance, therefore, necessitates
e-commerce values are decomposed into lower levels until a “succinct statement”
can be made about the value that e-commerce produces. According to the above theory
of decomposition, the value “should make sense to an interested observer”, i.e. those
seeking to justify the investment or performance. Decomposition therefore ends at the
most “primitive” level, which is achieved when a particular benefit has been singled
out and can be measured. At the lowest level, the value variable should have
“functional integrity” in the sense that the method of value measurement is achieved
through the application of a formulae or algorithm.
In adopting the approach to value decomposition, we identified the following
hierarchy. Value constructs are identified at the highest level. They are individually
decomposed into the next levels, which we refer to as value variables which are further
broken down into value metrics. The lower levels can be spliced back together to
reconstitute the high level value construct. Figure 1 shows the prosed value
decomposition hierarchy and splicing order.
Interpreting the theory of decomposition further shows that there are two major
challenges confronting the analyst:
(1) Proper splicing. This implies that values (contructs, variables and metrics)
should be carefully specified and that their interconnections should be
controlled. This is difficult to achieve since the specification of values at this

Value Value Value

Figure 1.
construct variables metrics Value decomposition
BIJ early stage of applying the theory will most likely be done in an arbitrary
13,1/2 manner. Furthermore, the interaction during decomposition becomes more
difficult at the lowest, metrics level. Constructs can be associated more readily
since they capture the broad value of an e-commerce application. However,
interfaces are more difficult to determine and control at the lower level where
the broad overview is being replaced by detail.
84 (2) Functional integrity. This implies that a succinct statement can be made for
each metric which makes sense to “an interested observer and should embody a
complete self-contained activity” as quoted above. In practice, of course, value
metrics can vary because it is determined from the perspective of the
stakeholder. Consensus on what the metric is intended to measure may be
difficult to achieve.

Decomposition and e-commerce performance

For the purpose of this paper we have envisaged e-commerce to comprise activities that
are internal/external to the business and focus on customer/business (Figure 2). For
example, B2C e-commerce is an activity primarily aimed at the external customer
environment. By contrast, enterprise resource planning is made up of internal systems
which connect mostly with other businesses. The applications that make up
e-commerce are as follows:
.
Visitor relationship management (VRM). The use of a web site to attract
customers to the organisation.
.
B2C e-commerce. The use of a web site to conduct electronic transactions
between the organisation and consumers.
. Business-to-business (B2B) e-commerce. The use of a web site to conduct
electronic transactions between the organisation and other organisations.
.
Customer relationship management (CRM). The management of the
organisation’s customers through a customer database and reporting software.
.
Electronic procurement (EP). The use of internet technology to conduct electronic
purchasing.
.
Enterprise resource planning (ERP). The management of the organisation’s
resources through integrated enterprise-wide software.

External Internal

* B2C
* VRM * CRM Customer
Integration
into
Business
* B2B
Figure 2.
E-commerce applications * EP * ERP Business
The paper will outline key value constructs, which are then decomposed into value E-commerce
variables, attached to each of the above applications (Tables I-VI). performance
Visitor relationship management
NetGensies (2000) conducted research among e-Businesses to establish insights into
measurements that are required to track the performance of web sites. They
acknowledge the difficulty of the task because of the current inability to identify proper 85
criteria and meaningful calculations which caused them to state:
Traditional management adage: you cannot manage what you do not measure; E-Business
addendum: you cannot measure what you do not define.
Dayal et al. (2000, p. 44) related the visitor’s experience with a web site to building
digital brands. They maintained that:
Digital brand builders should care about the consumer’s on-line experiences for the simple
reason that all of them – good, bad, or indifferent – influence consumer perceptions of a
product’s brand. To put it differently, on the Web, the experience is the brand.
Experience in the digital world can take many forms and include convenience,
achievement, fun and adventure, self-expression and recognition, and belonging.

Business-to-consumer
McDonald and Wilson (1999) define e-marketing as IT-enabled marketing through the
use of web-based technology. This creates e-marketing opportunities by affecting a
strategic change in interacting with customers, either directly or through a distribution
channel. First, the selling process can be improved by making the selling effort more
effective (e.g. by using the web to display products in a multi media form). Second,
customers’ buying and usage experiences can be enhanced. For the former it is possible
to provide services that make the product easier to buy (e.g. ordering it online through

Value construct Value variable

Stickiness Effectiveness of the web page to hold visitor’s

attention
Slipperiness Extent to which a visitor clicks though a banner/link
or web page
Focus Pages looked at as a proportion of total number of
pages
Velocity How quickly a visitor moves from purchase
awareness to purchase decision
Freshness Frequency in which web site is refreshed before the
visitor revisits the site
Seducible moments Junctures where a visitor is susceptible to an offer of
purchase
Abandonment Visitor commences purchase process but does not
complete it
Excitement Visitor having a satisfying experience when visiting
the web site
Counter Number of visitors that have made contact with the Table I.
web site Values of VRM
BIJ
Value construct Value variable
13,1/2
New customers The number and value of new customers acquired
through the web site
Existing customer The number and value of existing customers doing
business through the web site
86 Order delivery cycle time Reduction in delivery time to meet customer
demands due to the use of a web site
Service requests The number and value of service requests met and
resolved on a web site
Order fulfilment The extent to which orders are fulfilled to the
Table II. satisfaction of customers through the use of a web
Values of B2C site

Value construct Value variable

Costs Reduction in costs due to automation of high volume

transactions and data accuracy
Productivity Improved internal business efficiency through
automation of B2B transactions
Information exchange Improved information leading to better market
intelligence
Sources for new products Access to new products by being a link in a trading
community
Table III. Time-to-market Collaborative product design and supply chain
Values of B2B integration gets products to the market quicker

Value construct Value variable

Personalisaton Profile information of customers

Life time value The value obtained during the period of custom
Loyalty value The time they remain a customer
Attrition Reflects those who depart as customers and never to
return
Churn Number of customers lost over total number of
Table IV. customers
Values of CRM Feedback Good and bad feedback obtained from customers

the web); for the latter satisfaction of existing customers can be increased over the life
of their custom (e.g. through CRM).

Business-to-business
As Deise et al. (2000, p. 59) point out, historically companies have focused on
automating internal processes. However, web-based technologies have given rise to the
extraprise where businesses are linked for business communications and transactions.
Working within the extraprise, companies will develop business strategies with their
partners in mind. And a whole new breed of information systems will be created that
integrate the processes and information flow among partners.
 E-commerce
Value construct Value variable
performance
MRO Reduction in maintenance, repair and operating
(MRO) costs by avoiding wasteful and ad hoc
practices such as maverick buying
Interface Cost savings through the use of a web interface for
operational purchasing activities such as ordering, 87
expediting and administration
Sourcing Cost savings in tactical purchasing activities such as
setting specifications, selecting suppliers,
negotiations and contracting
Tendering As for sourcing, cost savings for tendering activities
Informing Cost savings in strategic purchasing activities such
as compiling expenditure analysis and developing Table V.
purchasing policies Values of EP

Value construct Value variable

Internal efficiency Streamlined inner workings of a business through

Business benefits
Interface with e-Commerce
Creating an extended enterprise
Application integration
Integration of front/back office
commonality and standardisation
Various benefits such as inventory reduction,
customer responsiveness, manufacturing flexibility
Automatic updating of purchases, sales, etc. without
the need for human involvement
Integration of core applications with those of
suppliers and customers
Integration of business activities to better leverage
information within the organisation
Linking external with internal activities through a
single coherent IT infrastructure and systems Table VI.
environment Values of ERP

This gives rise to values such as complementary services and products by working
together synergetically.

Customer relationship management

According to Schmitz (2000), CRM is a combination of business processes management
and IT applications to understand customers from a multifaceted perspective. The
organisation makes a strategic change by moving from transaction- to customer-based
marketing. Customer life time value (NetGenises, 2000; Rust et al., 2000) becomes an
important consideration as it recognises the shift from managing product profitability
to managing customer profitability. This relies on product promotions, providing the
right product for customer, offering price and convenience and customer service, and
acquiring the ability to identify, attract and retain the best customers. As a result
customer equity is build.

Electronic procurement
De Boer et al. (2002) define EP simple as the use of internet technology in the
purchasing process. The key benefits associated with EP arise from keeping the costs
BIJ of internal processes at a low level and increasing revenue-benefits through improved
13,1/2 product quality, innovation, etc. They identified different forms of EP including
electronic maintenance, repair and operating (e-MRO), e-sourcing, e-tendering and
e-informing. Each has its own benefits; for example, under e-MRO, purchasing
expenditures can be reduced by greater adherence to optimal company contracts which
avoids wasteful and ad hoc practices such as maverick buying.
88 EP is part of value chain management (Deise et al., 2000; Rayport and Sviokla, 1996)
where enabled organisations join together to deliver value to customers. Business
processes and information systems among customers, suppliers and other business
partners seamlessly transact business. This results in the elimination of no longer
required intermediaries, reduced inventory holding, and efficient distribution of
products and services.

Enterprise resource planning

The availability of ERP systems and the emergence of web-based technologies haven
given rise to the creation of the e-Business (Cameron, 2000; Violino, 2000; Wilder, 1999).
Enterprise-wide information systems and ITs such as data mining have provided
increased integration with internal as well as external systems leading to increased
flexibility and speed in meeting customer needs, e-commerce integration as well
improved management information.
Towards corporate performance management
With an increasing emphasis by organisations on speed of execution and market
responsiveness, metrics will increasingly be generated electronically. They are referred
to as e-metrics and appear to be gaining popularity because they can be generated
automatically from internet-based business services and are increasingly being
promoted by application service providers (Weischedel et al., 2003). They have the
benefit of collecting data about organisational performances in real-time and will
enable aggregate corporate performance management (CPM) performance measures to
be developed. CPM is about “connecting the dots” so that the various lower level value
metrics can be aggregated and made to work seamlessly together, reflecting the
highest level of corporate performance.
There are various organisational perspectives on what constitutes effective CPM.
The Gartner Group (Koulbanis, 2003) claims that their Business Performance
Frameworke holistically covers the controllable activities that occur within the typical
organisation, namely demand management, supply management and support services.
They are further broken down into what the Gartner Group refers to as aggregate
measures (“value variables” in this paper) as shown in Table VII.

Value construct Value variable

Demand management Consisting of market responsiveness, sales

effectiveness and product development effectiveness
Supply management Consisting of customer responsiveness, supplier
effectiveness, operational effectiveness
Support services Consisting of human resource responsiveness, IT
Table VII. responsiveness and finance and regulatory
Values of CPM responsiveness
Value metrics E-commerce
It is obvious from this paper that there is large agreement on the key values obtained performance
from e-commerce because they are reflected in the majority of published material.
However, it is difficult to achieve the degree of integrity required by the decomposition
technique at the metrics level. As indicated earlier in the paper there are two key
challenges to be met, namely achieving value splicing and functional integrity.
Among the organisations that have developed e-metrics is NetGenises (2000) who 89
have developed models down to the metric level from interviews conducted with CIOs.
For example, the metric for the value construct of stickiness for a VRM system is given
as follows:
Stickiness ¼ Frequency £ Duration £ Total site reach
where
Number of visits in time period t
Frequency ¼
Number of unique users who visited in t
and
Total amount of time spent viewing all pages
Duration ¼
Number of visits in time period t
Taking the above example of VRM further, raises a number of issues. First, metrics
undergo changes over time. As Weischedel et al. (2003) found, during the 1990s, web
traffic measures were mainly used but they have now been superseded by more
sophisticated metrics such as the one above, customer re-activation and customer
loyalty. The metrics are complied using web analysis and reporting software.
However, concern has been found about the reliability of these e-metrics. For example,
the use of server caches which store web pages distorts the number of requests to web
pages on the server when the same page is viewed again.
While the approach to metrics can have an empowering influence it can be
threatening to some. There is now an expectation that metrics will me met. There could
be different reactions to this, for example, some may be reluctant to commit to
achieving targets that they perceive to be unreasonable or even exhibit
resistance-to-change attitudes. There could also be undesirable behaviour such as
seizing control over other business activities or using information as a means to exert
broader influence in the organisation.
There are, however, a number of key facilitators to the acceptance of the use of the
performance metrics approach. The most significant enabler is the existing business
climate requiring today’s organisation to be lean and agile, and adaptive to a rapidly
changing environment. Metrics enable them to track their internal as well as external
effectiveness. Second, the development of cross-functional metrics, in preference to a
silo approach, has increased the level of communication within business, thereby
providing the environment to engage with each other across business disciplines.
Furthermore, metrics have the potential to facilitate business integration and deliver an
integrated value framework for the organisation.
As stated earlier in the paper, the decomposition approach requires proper splicing.
This should take place as values are decomposed at each level, from value construct, to
BIJ value variable, to value metric. It is at the last level that this requirement becomes most
13,1/2 critical since this is where functional integrity is achieved. As discussed above, values
are developed at two levels – CPM at the top level of the organisation and the
e-commerce systems at the lower levels. Value metrics for each of them will have to be
integrated so as to provide a complete value framework for the organisation. The
framework is shown in Figure 3.
90
Conclusions and further research
It is recognised that further research is needed to increase the knowledge base of what
actually takes place in practice. For example, base line research should be conducted in
which organisations are surveyed as to their knowledge of metrics and the perceived
need for metrics. However, from the rapidly increasing interest in IT evaluation,
reflected in publications and conferences, one can assume that the need for
performance measurement is well accepted in practice.
As far as theory and technique developments are concerned, research in this respect
is currently lacking. This paper attempts to explore the basis for which the approach of
decomposition can provide insight and rigour to measuring modern IT values. As
such, the paper provides a discussion of the decomposition technique as applied to
e-commerce performance measurement. It has demonstrated that is readily possible to
identify, at the highest level of value decomposition, constructs and variables. A review
of a relatively small amount of literature indicated the ease with which constructs and
variables could be defined for the key e-commerce applications. However, far less
agreement and information is available on what are appropriate metrics.
There are a number of research challenges. The first is to develop theoretical models
that satisfy the generic requirements of e-commerce applications. An example of the
approach is that by NetGenises (2000). However, current empirical research is limited
and far more extensive research would have to be conducted to achieve acceptance for
such value models. Second, tools and techniques need to be provided to business

Corporate Performance Management

Demand Supply Support Value
management management management constructs
Market, sales, product Customer, suppliers, Human resource, IT, Value
development operations finance, regulations variables
Value
metrics

Metrics Integration
Value
metrics
Visitors’ Value of Reduced Profile Reduced Streamline Value
att’n, etc custom, etc costs, etc info, etc costs, etc ops, etc variable
Stickiness Customer, Costs, etc Personali- MRO, etc Efficiency Value
Figure 3. etc etc sation, etc etc constructs
An integrated value
framework VRM B2C B2B CRM EP ERP
analysts that would help them to identify and decompose values within the E-commerce
organisation. This may a more realistic approach than developing generic models since performance
it is unlikely that standardisation in e-commerce applications and hence values can be
achieved. Furthermore, the approach of providing techniques and tools has worked
well in the discipline of systems engineering where business analysts had to analyse
many diverse business settings. Third, the requirement of splicing needs to be satisfied
at all levels of value decomposition. This paper points to a starting point at the level 91
where CPM and e-commerce metrics integrate.

References
Cameron, P. (2000), “Measuring up”, CMA Management, March.
Davis, W.S. (1994), Business Systems Analysis and Design, Wadsworth Publishing Company,
Belmont, CA.
Dayal, S., Landesberg, H. and Zeisser, M. (2000), “Building digital brands”, The McKinsey
Quarterly, available at: www.mckinseyquarterly.com/electron/budi00.asp
De Boer, L., Harink, J. and Heijboer, G. (2002), “A conceptual model for assessing the impact of
electronic procurement”, European Journal of Purchasing & Supply Management, Vol. 8,
pp. 25-33.
Deise, M.V., Nowikow, C., King, P. and Wright, A. (2000), Executive’s Guide to E-Business: From
Tactics to Strategy, Wiley/PriceWaterhouseCoopers, New York, NY.
Donovan, R.M. (n.d.), “Why the controversy over ROI from ERP?”, available at: www.
rmdonovan.com/pdf/perform.pdf
Eliason, A.L. (1987), Systems Development Analysis, Design, and Implementation, Little, Brown
and Company, Boston, MA.
Feeney, D. (2001), “Making business sense of the e-opportunity”, MIT Sloan Management
Review, pp. 41-51.
Gillettt, W.D. and Pollack, S.V. (1982), An Introduction to Engineered Software, Holt, Rinehart
and Winston, New York, NY.
Gouscos, D., Kalikakis, M., Legal, M., Papadopoulou, S. and Verginadis, G. (2003), “A
performance and quality assessment model for one-stop government-to-business
e-services”, Proceedings of Business Excellence: Performance Measures, Benchmarking
and Best Practices in New Economy, University of Minho, Portugal.
Hawryszkiewycz, I.T. (1994), Introduction to Systems Analysis and Design, Prentice-Hall, Sydney.
Koulbanis, N. (2003), “Total value of opportunity: using business metrics to shed light on IT
investments”, Proceedings of Gartner Symposium, Sydney, Australia.
McDonald, M. and Wilson, H. (1999), e-Marketing: Improving Marketing Effectiveness in a Digital
World, Financial Times/Prentice-Hall/Cranfield University, School of Management, New
York, NY/Cranfield.
NetGenises Corp. (2000), “E-metrics: business metrics for the new economy”, White paper,
available at: www.netgen.com/downloads/pdf/emetrics/e-metrics_business_metrics_for
_the_new_economy.pdf
Rayport, J.F. and Sviokla, J.J. (1996), “Exploiting the virtual chain”, The McKinsey Quarterly,
Vol. 1, pp. 21-37.
Rust, R.T., Zeithaml, V.A. and Lemon, K.N. (2000), Driving Customer Equity: How Lifetime
Customer Value is Reshaping Corporate Strategy, Free Press, New York, NY.
BIJ Schmitz, P.F. (2000), “The art and science of measuring the business value of CRM”, available at:
www.gartner.com
13,1/2 Violino, B. (2000), “Payback time for e-business”, Information Week, available at: www.
internetwk.com/lead/lead042800.htm
Weischedel, B., Matear, S. and Deans, K.R. (2003), “The use of e-metrics in strategic marketing
decisions – a preliminary investigation”, Proceedings of Business Excellence: Performance
92 Measures, Benchmarking and Best Practices in New Economy, University of Minho,
Portugal.
Wilder, C. (1999), “E-business: strategic investment”, Information Week, available at: www.
informationweek.com/735/roi2.htm

Corresponding author
Dieter Fink can be contacted at: d.fink@cowan.edu.au

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

Action research
Action research and networking and networking
benchmarking in developing benchmarking
Nordic statistics on woman
93
entrepreneurship
Paula Kyrö
Research Centre for Vocational Education, University of Tampere,
Hämeenlinna, Finland

Abstract
Purpose – The purpose of this study is to participate in the current debate on developing the
theoretical basis for the concept and process of benchmarking. The paper aims to explore the prospects
for a new form of benchmarking – networking benchmarking – and adopts, as the scientific
framework of the process, an action research approach that combines scientific method and practical
developmental work.
Design/methodology/approach – In order to explore the possibilities of these new theoretical
constructions, this study applies them in a Nordic project, in which the representatives of five Nordic
countries studied together how to develop their national statistical procedures on woman
entrepreneurship by benchmarking and developing best practices for use in the Nordic countries.
Findings – The results supported the idea that benchmarking is a special kind of action research.
The results also indicated that the need for explicating and mutually sharing adopted theoretical
frames turned out to be one of the key points in increasing the efficiency and innovativeness of the
process. Consequently, networking benchmarking and the action research model indeed have potential
both as a theoretical frame and a practical tool, thus encouraging their further development.
Research limitations/implications – It should be noticed, however, that the present study is only
a very minor effort and an example of the possibilities scientific knowledge can offer to the
developments of benchmarking.
Practical implications – From the practical perspective as a whole, this study offers a large range
of opportunities for Nordic co-operation within the field of entrepreneurship data gathering.
Originality/value – Considering more specifically future research directions from this study, the
results encourage studying both the networking benchmarking the action research approach in
different contexts. On the other hand, the findings also give some ideas for further development of the
action research model.
Keywords Benchmarking, Action research, Networking, Entrepreneurialism, Women, Statistics
Paper type Research paper

Introduction
Benchmarking has established its position as a tool to improve an organization’s
performance and competitiveness in business life. Recently, the debate on the need to
conceptualise and revise both the concept and process of benchmarking has
strengthened (Longbottom, 2000; Yasin, 2002). This has taken place together with its Benchmarking: An International
Journal
Vol. 13 No. 1/2, 2006
The author is indebted to the Jönköping International Business School for providing the excellent pp. 93-105
q Emerald Group Publishing Limited
opportunity, databases and other facilities for this study and to the whole Nordic network of this 1463-5771
project for a challenging process. DOI 10.1108/14635770610644600
BIJ expanding scope. Recent studies indicate that targets for its application have extended
13,1/2 from large firms to small businesses as well as the public and semi-public sectors, and
from single firm towards network structures (Ball, 2000; Davis, 1998; Jones, 1999;
Kulmala, 1999). Bhutta and Huq (1999) also claim that the current tendency is to view
benchmarking as a continuous development process rather than a problem-solving
project.
94 The conceptual study for the revision of the concept of benchmarking suggested
that, in order to meet future challenges for innovative practices and diverse contexts of
benchmarking, there is a need to study and further develop a new form of networking
benchmarking. The same study also argued that there is a need to find scientific
methods for studying the actual process of benchmarking (Kyrö, 2003). A more recent
study further elaborated this suggestion and applied an action research approach in
order to provide a method for explicating how the process of benchmarking leads to
intentional and transferable results for identifying and creating new best practices
(Kyrö, 2004). However, even though the results of the study were promising, they were
still very preliminary in theorising the benchmarking process and thus open to critics
from various directions. Above all, there is a need for empirical studies about testing
the validity of the action research approach.
Related to both of these challenges – conceptual and methodological – this
study combines the concept of networking benchmarking to the action research
approach and explores their prospects to provide together a scientific framework for
conducting a benchmarking process. As an empirical context, this study applies
them in the case of the Nordic project, in which the representatives of five Nordic
countries studied together how to develop their national statistical procedures on
woman entrepreneurship. It was assumed that this complex and altogether
four-year-long process offered an opportunity to test and further develop this new
combination of the conceptual and process frameworks. Thus this study participates
in the current debate on developing the theoretical basis for the concept and process
of benchmarking.

Networking benchmarking
Even though the definitions of benchmarking vary between scholars, the aspects of
evaluation and improvement by learning from others are embedded in the different
definitions regardless of the definer (Ball, 2000; Büyüközkan and Maire, 1998;
Carpinetti and de Melo, 2002; Comm and Mathaisel, 2000; Elmuti and Kathawala, 1997;
Fernandez et al., 2001; Prado, 2001; Watson, 1993; Yasin, 2002; Zairi and Whymark,
2000).
The nature of learning, however, has recently changed. Previously, the focus was on
model learning, i.e. learning from others, but towards the end of twentieth century, it
was accompanied by learning with others.
The advantages of networking and learning with others embedded in it can be
reflected through the criticism that model learning has received. Longbottom (2000)
raises the dilemma arising between copying competitors and gaining competitive
advantage through distinctive performance. On the other hand, Davis (1998) proposes
that especially in the public sector, instead of benchmarking antiquated practices, it
would be better to just invent new ones. Another argument against model learning
relates to the changing environment (Hammer and Champhy, 1993). The idea is that
imitating existing practices is too slow and incremental. Consequently, there is a need Action research
for faster and more radical approaches. and networking
The advantages of benchmarking by networking are apparent in public sector
organisations. The basic nature of public services is not to compete with each other; benchmarking
rather, they have been established in order to provide the best possible services as
effectively and efficiently as possible. If one organization succeeds in providing
excellent solutions, it is understood that these are open for others to use as well. The 95
focus is more on cooperation than on competition. Of course, this becomes more
complicated when public and private sectors’ practices are intermingled and when, for
example, they offer similar services with genuine competition (Kyrö, 2003).
The need to create future-oriented solutions and processes that cannot be found
from existing practices is also international and global, since on the one hand, mutual
problems and goals create various alliances between different nations and, on the other
hand, these problems often relate to global developments rather than to an individual
nation or firm. Contemporary competition is not strongly focused, but rather there are
mutual problems for the future. This motivates and allows experience sharing in
networks. An example of this is Prado’s (2001) article about Spanish companies,
describing business networking for experience sharing in quality. The companies had
a mutual problem that motivated them to create new solutions in a network.
On the other hand, networks have also established themselves as a business
structure on local, regional, national as well as international levels. Instead of involving
a single unit or organization, benchmarking could take as its starting point a network
on both sides as participants and as partners.
Thus the importance sharing experiences and creating solutions in networks has
increased due to:
.
the need for generative, future-oriented solutions mutual to several partners;
.
the emergence in benchmarking of public sector organizations with their specific
problems;
.
the increasing impact of global developments on cooperation and the nature of
the problems; and
.
the structural changes in business as well as other organizations.

The demand to explicate networking benchmarking as a new form in this field reflects
these challenges. This means that the benchmarkers and the partners against whom to
benchmark comprise the same network. Its special feature is that partners learn
together and from each other in order to create new practices, rather than learn and
apply existing best practices created by others.
Thus networking benchmarking refers to a form of benchmarking, in which the
partners and benchmarkers are the same, with mutual motives and/or problems and
with the aim of learning together from and with each other in order to create new best
practices. As it is a new form of benchmarking, it also expands the scope and concept
of benchmarking.

Action research approach

Action research aims to study and influence practices in social movements and work
communities as an interplay between science and practice. Suojanen (2001) specifies its
BIJ aims as two different trends, an education-oriented trend focusing on improving
13,1/2 learning processes and competencies, and a project-oriented trend emphasising
developing and improving a specific practice other than learning itself, such as product
development, customer service, etc.
The findings of a conceptual study (Kyrö, 2004) comparing these trends, features
and cyclical phases of action research to their counterparts in benchmarking indicated
96 that:
.
Their aims were similar, but benchmarking also had a specific feature relating to
the concept of “best practice”. In action research, any change that improved
practices within an organisation might be satisfactory, regardless of its
excellence compared to the possibilities offered by outside actors and the
environment.
.
Actually, the benchmarking process had indeed an apparent resemblance to
action research, assuming that it explicated its phases according to action
research phases and cycles as well as focused more on theoretical frames,
observation and reflection, all of which are essential for distinguishing a
scientifically conducted action research process from practical work.

As a base for comparison, the study adopted Suojanen’s two-cycle action research
model. The model starts by analysing the actual situation and envisioning a course of
action. This is followed by plan, action, observation and reflection. Reflection leads to a
revised plan followed by a new action research circle (Suojanen, 2001).
In a benchmarking process, the actual data collection phase is regarded as an action
of the first cycle, while implementation refers to the second cycle.
In order to explore these findings in the practical context, this study applies the
model to the Nordic networking benchmarking project that studied and developed
statistics on woman entrepreneurship.

The Nordic project

Participants and aims of the project
The benchmarking project “woman entrepreneurship in the Nordic countries” started
in 1998 from an SMB-Forum initiative, and the first action research cycle was
completed in June 2001. The SMB-Forum operates under the Nordic Industrial Fund
(NIF) that is subsidiary to the Nordic Council of Ministers.
The original purpose was to define the conditions faced by women entrepreneurs
and their role in the economies of the Nordic countries. Thus the participants shared a
mutual aim and vision of ensuring and improving women’s equal position in the
economy.
At the beginning, it was obvious that benchmarking took place as a network
consisting of the NIF and all five Nordic countries – Denmark, Finland, Iceland, Norway
and Sweden. Country representation consisted of the representatives of SMB-Forum’s
work group of “women’s role in business life” and later also included a researcher from
each country. These were further complemented with national statistical experts.
The strategy of the network project was originally to contribute to knowledge
sharing at the Nordic level, to learn together and from each other as well as to ensure
optimal exploitation of research results. It turned out, however, that besides learning
from each other, there was a need to create new practices together. The participants
also wished to attain excellence at a Nordic level, rather than compete with each other. Action research
Since comparability is the mutual aim of these countries, co-operation, not competition and networking
is the essence of the benchmarking. Thus the starting point was to learn in a network
in which both the partners and the benchmarkers were same actors. benchmarking
Thus the project fulfilled all three criteria for networking benchmarking. The
partners and benchmarkers were the same, having a mutual motive and aim to learn
together from and with each other in order to create new best practices. 97
The phases of the benchmarking process
The phases of the project (Figure 1) can be divided into two cycles. The first cycle
started with clarification of the actual situation, proceeding to a more accurate
plan, then to data gathering and analysis as an action phase; these were
accompanied and partly followed by observation and reflection phases. The revised
plan starting the next cycle concerned the country-specific implementation; that is
still going on.
The current situation and the vision for the future – 1998. The project started in
1998 when the SMB-Forum established a work group on “women’s role in business
life”. For Nordic societies, women entrepreneurship and knowledge about it is an
essential issue in working life and in welfare discussion. This relates to the strategic
competencies of the Nordic countries. It was, in fact, assumed that this project should
facilitate in general the individual countries’ efforts to stimulate the small business
sector.
Each country invited one representative to work in this group. These five
representatives prepared preliminary country reports on the status of women’s
position in business life in their country. These reports indicated that each country had
difficulties in presenting the conditions that women entrepreneurs face and their role in
the economies. This related partly to difficulties in presenting sufficiently adequate
statistics on woman entrepreneurship. The need for statistics on woman
entrepreneurship and, more generally, for gendered statistics was also identified as
a world-class problem (Holmqvist, 1996; OECD, 2001). This need reflects the strategic
changes in the structures of economies.
Owing to these jointly identified, Nordic, as well as global, difficulties the network
decided to concentrate on examining the available gender-oriented official statistics of
woman entrepreneurship at the Nordic level and benchmark best practices related to it.
Thus the project was rather project- than learning-oriented.
Planning. During 1999, two workshops prepared a project proposal that specified
the target and the resources needed, and also prepared the time schedule and data
collection plan. In the proposal, the project was planned to proceed in four phases:
preparation in spring 2000, data gathering between March and April 2000, discussion
of the final results and preparation of the final report between May and September
2000 and delivering the final report with recommendations in February 2001. In
practice, the data gathering proved to be far more complex than expected.
Consequently, the schedule was extended to June 2001.
As a theoretical framework, the list of desired data was constructed according to the
life cycle from intentions to the start-up-phase, then on to existing businesses and
finally to expiring. Emphasis was also put on the data-gathering sources, processes
and plans for future development. Finally, the plan was supplemented by the
BIJ The spiral of action research
13,1/2
1998
Clarification of the
actual situation
- preliminary country studies
98 - results: difficulties to provide
gendered statistics

1999
2001

Plan
Retro- and
proactive reflection
Reflection
2001-2002 Reflection
Comparison Observation
and results Action
-choosing best
practises Observation Action
-took place in
several phases 2000-2001
-real-time, retro- Data gathering and
Revised plan

and proactive Reflection

reflection
analysing
-innovative process
-took place in several
phases

Observation Action 2001

Country-specific
plans for applying
best practices
Improved action -developing
practise

Figure 1.
The Nordic project in the Source: Basic model modified from
action research spiral Soujanen 1998

organizational structures of data gathering. It was assumed that each country would
find out how its data gathering was organized, and what statistics were available and
that they would also deliver the actual figures. The aim was to analyse and define the
practices, methods and preconditions behind the actual performance, as process
benchmarking requires (Kulmala, 1999).
 Action. Data gathering proceeded as a process during 2000-2001. Analysis also took Action research
place as a process consequent to the data deliveries. and networking
The first analysis process indicated that, even though there was a detailed
data-gathering plan that was also discussed in several working meetings, it was still benchmarking
hard to get consistent descriptions of the actual statistical processes. The process of
providing statistics was, therefore, further specified using detailed flow charts as
guidance. As another frame of reference, the flow chart helped in gaining a deeper level 99
of analysis. Instead of merely focusing on the outcomes, it also provided keys for
comparing the actual processes giving rise to the statistics. It thus helped in choosing
and developing the best practices, as well as making it possible to apply these
practices. Hence it provided tools for proceeding thereafter to the country-specified
applications.
The analyses revealed that countries’ abilities to provide gendered statistics varied
considerably. None of the countries could generate sufficient data on women’s business
activities. Rather, they were able to provide some solutions for specific problems within
this field. Also none of the Nordic countries had identified or conceptualised those
persons having intentions of starting their own businesses. The results indicated that
the problem is not only a lack of knowledge as such, but that it lies too in the
comparability of the existing data, due to the different production processes and their
shortcomings in combining individuals and their businesses. Thus, considering the
delivered figures, it was not possible to make comparisons between countries, but
rather list what figures were available.
The analyses of the definitions deepened these findings. International debate on the
concept of entrepreneurship offered a scientific frame of reference for these analyses.
Based on it three concept categories were identified:
(1) new economic activity combining ownership and management;
(2) economic activity combining ownership and management; and
(3) self-employed and family members.

Only Sweden could provide adequate data in the first category with gender criteria.
Denmark could generate it from personally owned enterprises. In the second category
none of the countries provided regularly gathered data. All countries were able to
provide gendered statistics in the third category, due to the Labour force statistics.
Even in this category the register-based data gathering in Denmark provided
somewhat different results compared to other countries. An extra difficulty was due to
the different industry and legal-form coverage as well as differences in turnover
requirements.
When it turned out that none of the countries was actually able to solve the essential
problem of combining the management and ownership of existing firms, the project
was extended to looking for genuinely new solutions. Norway made a pilot study that
combined these from official registers. Denmark also offered a large study that
contained examples and elements of further developing statistical procedures in this
respect.
Choosing best practices. Both the life-cycle model and the identified categories
formed the criteria for creating and choosing the best practices. Proposals concentrated
also on possible solutions to the identified problems. For the starting phase in the
definition category of entrepreneurship as a new economic activity, Sweden provided
BIJ regularly gathered gendered statistics on newly started enterprises. This solved the
13,1/2 problems of combining management and ownership and handling the newness of the
enterprise, as well as offering the possibility of describing individuals’ own decisions
and motives. However, for future needs and for applications in other countries, it was
recommended that the questions concerning the motives of women entrepreneurs
should be updated according to the most recent research findings. Denmark’s extensive
100 study gave examples of that.
Denmark also came up with a register-based solution for partly identifying the
newness of business activity.
For the existing phase in the category of entrepreneurship as an economic activity
combining ownership and management, the Norwegian pilot study gave a solution for
combining individuals and firms.
In the third category of self-employed, the Finnish example, for evaluating and
improving the flexibility and, complexity of the categorisation, offered an alternative
for improving data gathering.
For future developments, there is also a need to find solutions for getting data about
intentions as well as to harmonise the terminology used, in order to avoid unnecessary
misinterpretations and to facilitate international comparisons.
Observation. Planned observation most clearly distinguishes action research from
practical work, and is concerned with how to gather data and analyse them during the
reflection process. In this case Finland co-ordinated data gathering as well as analysed
and reported the process. Each country, however, was responsible for country-specific
reports. Reporting took place in several phases according to data deliveries. The final
reports were also published for further reflection and for implementation (Kyrö, 2002a,
b; Kyrö and Tyrväinen, 2002).
In this study, reporting was rather a continuous process and a tool for reflection,
rather than a phase following the observations, as the cyclical model assumes.
Reflection. Reflection is a means of finding out how the action research has
succeeded in developing both the studied object and the learning process. Its aim is to
lead to a revised plan for the next cycle. Action research includes real-time, retro- and
pro-active reflection. Reflection consists of three levels, technical, practical and critical
(Suojanen, 2001).
It should be mentioned, however, that in this study the reflection was a continuous
process, rather than being the final phase of the cycle. In this report, therefore, the
chosen best practices follow immediately on the action phase. The interplay between
action, observation and reflection generated new solutions. Perhaps this is typical of
networking benchmarking with its aim of creating new innovative solutions. It is not
possible without continuous reflective learning. Even so, there is always a need to
engage in retro-, and pro-active reflection before entering into a new cycle.
At the technical level, the reflection concentrates on describing activity, and
evaluating models and objectives (Suojanen, 2001). This project was an example of a
new form of networking benchmarking in which the benchmarkers and partners are
the same. They learned from each other and together developed new innovative
solutions. The results, i.e. the chosen best practices and the recommendations relating
to them, made it clear that all countries could benefit from the project.
The analyses gave explicit directions to each country for national applications and
also revealed those areas in which mutual Nordic development is necessary. The study
also revealed the gaps that are to be filled for Nordic comparisons to be made possible. Action research
Thus the project specified the need for a revised plan that would lead to the next action and networking
research cycle. For that purpose, participants also prepared a plan for sharing the
knowledge with different interest groups. benchmarking
Practical level reflection focuses on the experiences and the basis for choices
(Suojanen, 2001).
In this project, the extra challenge of networking consisted in its composition. The 101
core team of the country representatives was supplemented with researchers. In
practice it turned out that these teams were very flexible, and there were several
changes in team members during the process. The final team of researchers covering
all countries was only collected at the very end of the process. This, together with the
flexible aims in the planning phase, increased still further the complexity of the
process.
Additionally, it was not obvious at the beginning that the bases for the choices
made were always logical and solidly argued for. Employing an explicitly scientific
frame of references at the very beginning might have improved and speeded up the
process. In this respect, the shortcomings in the use of theoretical frames identified in
the conceptual study turned out to be a critical issue in this study.
However, the various different insights into the phenomenon clearly made a positive
contribution to the quality of this project and also enlarged the circle of
knowledge-sharing. Improving the quality of statistics is not only a matter of
statistical expertise, but rather of co-operation between different interest groups and
different experts and authorities. Therefore, experiences gained during this project
might also be beneficial in the future.
Critical level reflection also considers the values and, ways of working and thinking,
and, at best, provides an organisation’s collective memory with, explicit models for
coping with change processes, i.e. models of how to learn (Suojanen, 2001).
This project taught how important a mutually-shared understanding of the aims
and the process is. This problem was met with in this research in two respects. First,
even though all decisions were mutually accepted, and the data needed as well as the
time schedules were discussed together, the national reports varied quite considerably,
reflecting different commitments and different ways of thinking. This indicated how
important it is in networking benchmarking to reserve time and other resources for
mutual interaction between all participants and in all phases of the process.
Second, even though the participants shared the ultimate goal, and belief in the
value, of gaining more gendered knowledge on entrepreneurship, and jointly accepted
the plan for the project, the focus of motives and roles in the process varied. The
workload was thus unevenly divided between the participants. This is perhaps an
unsolved problem in processes like this. Dividing the workload more evenly between
participants might increase inconsistency even more than in this project. However,
consideration of this aspect at the beginning of the process, as well as better real-time
reflection during the process, might solve part of this problem.
Next cycle – revised plan. Since the development of data gathering requires the
assessment and prioritisation of different aspects, each country has to decide what is
wanted and what is regarded as valid. Therefore, the next cycle concerns specific
national applications. Revised plans and implementation of the results have started to
actualise. The Ministry of Trade and Industry in Finland has made a detailed plan for
BIJ developing gendered statistics according to the recommendations made in this study.
13,1/2 Denmark also stated that, encouraged by this study, they are considering the further
development of their concepts and a repeat of their extensive study. Norway has also
indicated its willingness to develop entrepreneurship data gathering. In Iceland
development is already an on-going process. Owing to the first cycle, all countries now
have the opportunity of following more consistently the efforts and practices in the
102 other Nordic countries and to learn from the experiences gained during this first cycle.

Summary and conclusions

The aim of this study was, on the one hand, to explore the prospects of the new concept
of networking benchmarking and, on the other hand, to apply the action research
approach to a benchmarking process. It was assumed that the need for future-oriented
solutions mutual to several partners and the special needs of public sector
organisations required innovative and collaborative networking, rather than copying
old practices from others.
Applying these ideas to the case of the Nordic data-gathering project indicated that
networking benchmarking model helped in explicating existing practices and their
advantages at the Nordic level, thus encouraging knowledge sharing for developing
coherent and comparable data gathering. In addition, it turned out to be a form of
benchmarking that generated new solutions. Both of these also turned out to be valid
for solving problems of gathering gendered knowledge of entrepreneurship at a global
level. Thus, the chosen networking strategy for contributing to knowledge sharing at
the Nordic level, to learn together and from each other and to attain excellence at the
Nordic level, rather than compete with each other turned out to be possible through
networking benchmarking. This encourages studying and developing this new form of
benchmarking further in different contexts with diverse participants and problems.
Applying the cyclical model of the action research approach as the research method
and a practical tool to conduct the process provided several aspects to be considered in
the future.
It was assumed that, besides an apparent resemblance to the cyclical action research
process, benchmarking had a specific feature of focusing only on “best practices” and
ignoring other changes. The results of this specific case indicated that this demand was
attainable. Thus, it supported the idea that benchmarking is a special kind of action
research.
It was also suggested that to distinguish a scientifically conducted action research
process from practical work requires addressing more focus on theoretical frames,
observation and reflection. The results indicated that the need for explicating and
mutually sharing adopted theoretical frames turned out to be one of the key points in
increasing the efficiency and innovativeness of the process. Thus, reserving resources
and time for those is a critical point in a successful process.
Considering the phases of observation and reflection, the results not only verified
their critical role in the process, but also brought some new insights into the nature of
these phases. In this study, reporting was more of a continuous process and a tool for
reflection than a phase following the observations, as is assumed in the cyclical model.
Also, in this study reflection was a continuous process, rather than being the final
phase of the cycle. It actually turned out that recognising opportunities and conditions
for new practices required constant interplay between action, observation and
reflection during the cycle. This indicates that retro- and pro-active reflection before Action research
entering into a new cycle should be complemented with continuous real-time retro- and and networking
pro-active reflection within each cycle. This is delineated by the inner circles in the first
cycle in Figure 1. This encourages to suggest that the creation of new practices seem to benchmarking
require continuous cycles of observation and reflection.

Evaluation and implications 103

This study focused on advancing the current debate on developing the theoretical basis
for the concept and process of benchmarking. The interwoven combination of
networking benchmarking and action research offered some new ideas to this
ambitious goal. The results encourage strengthening the interplay between science and
practice, since the intentional use of conceptual and methodological scientific frames of
references seem to increase the efficiency and innovativeness of the process. This
proportions the results of this study to the future expectations in this field indicating
that the present study is only a very minor effort and an example of the possibilities
scientific knowledge can offer to the developments of benchmarking. However, the
most promising directions might be found from those fields of science that study the
dynamics of learning and managements, i.e. the process of creating and applying best
practices.
Considering more specifically future research directions from this study, the results
encourage studying both the networking benchmarking and Suojanen’s cyclical model
of the action research approach in different contexts. On the other hand, the findings
indicate that the phases of the action research cycle are perhaps overlapping rather
than chronological, thus giving some ideas for further development of the action
research model.
From the practical perspective as a whole, this study offers a large range of
opportunities for Nordic co-operation within the field of entrepreneurship data
gathering. It also shows the mutual and national shortcomings in offering the best
practices for different phases in the life cycle. The findings also offer solutions to other
countries in a similar situation. Since the problem of gathering gendered data on
entrepreneurship is a global problem, these Nordic examples might offer seeds for
sharing knowledge at the global level.

References
Ball, A. (2000), “Benchmarking in local government under a central government agenda”,
Benchmarking: An International Journal, Vol. 7 No. 1, pp. 20-34.
Bhutta, K.S. and Huq, F. (1999), “Benchmarking – best practices: an integrated approach”,
Benchmarking: An International Journal, Vol. 6 No. 3, pp. 254-68.
Büyüközkan, G. and Maire, J-L. (1998), “Benchmarking process formalization and a case study”,
Benchmarking for Quality Management & Technology, Vol. 5 No. 2, pp. 101-25.
Carpinetti, L.C.R. and de Melo, A.M. (2002), “What to benchmark? A systematic approach and
cases”, Benchmarking: An International Journal, Vol. 9 No. 3, pp. 244-55.
Comm, C.L. and Mathaisel, D.F.X. (2000), “A paradigm for benchmarking lean initiatives for
quality improvement”, Benchmarking: An International Journal, Vol. 7 No. 2, pp. 118-27.
Davis, P. (1998), “The burgeoning of benchmarking in British local government”, Benchmarking
for Quality Management & Technology, Vol. 5 No. 4, pp. 260-70.
BIJ Elmuti, D. and Kathawala, Y. (1997), “An overview of benchmarking process: a tool for
continuous improvement and competitive advantage”, Benchmarking for Quality
13,1/2 Management & Technology, Vol. 4 No. 4, pp. 229-43.
Fernandez, P., McCarthy, P.F. and Rakotobe-Joel, T. (2001), “An evolutionary approach to
benchmarking”, Benchmarking: An International Journal, Vol. 8 No. 4, pp. 281-305.
Hammer, M. and Champhy, J. (1993), Re-engineering the Corporation: A Manifesto for Business
104 Revolution, Brealey, London.
Holmqvist, K. (1996), The Female Entrepreneur – Woman and/or Entrepreneur? Aspects of
Women Entrepreneurship, Swedish National Board for Industrial and Technical
Development, Stockholm, B 10, pp. 87-115.
Jones, R. (1999), “The role of benchmarking within the cultural reform journey of an
award-winning Australian local authority”, Benchmarking: An International Journal,
Vol. 6 No. 4, pp. 338-49.
Kulmala, J. (1999), “Benchmarking ammatillisen aikuiskoulutuskeskuksen toiminnan
kehittämisen välineenä”, Acta Universitatis Tamperensis 663, Tampere.
Kyrö, P. (Ed.) (2002a), “Woman entrepreneurship in the Nordic countries”, Country Reports:
Denmark, Finland, Iceland, Norway, Sweden. University of Jyväskylä, Reports from the
School of Business and Economics, Vol. II No. 27.
Kyrö, P. (2002b), “Benchmarking Nordic statistics on woman entrepreneurship”, JIBS Research
Reports Jönköping International Business School, No. 3.
Kyrö, P. (2003), “Revising the concept and forms of benchmarking”, Benchmarking: An
International Journal, Vol. 10 No. 3.
Kyrö, P. (2004), “Benchmarking as an action research process”, Benchmarking: An International
Journal, Vol. 11 No. 1.
Kyrö, P. and Tyrväinen, P. (2002), “Woman entrepreneurship in the Nordic countries”, Reports
from the School of Business and Economics, Vol. I No. 26, University of Jyväskylä.
Longbottom, D. (2000), “Benchmarking in the UK: an empirical study of practitioners and
academics”, Benchmarking: An International Journal, Vol. 7 No. 2, pp. 98-117.
OECD (2001), Women Entrepreneurs in SMEs. Realising the Benefits of Globalisation and the
Knowledge-Base Economy, OECD, Paris.
Prado, J.C.P. (2001), “Benchmarking for the development of quality assurance systems”,
Benchmarking: An International Journal, Vol. 8 No. 1, pp. 62-9.
Suojanen, U. (2001), “Action research”, available at: www.metodix.com
Watson, G.H. (1993), Strategic Benchmarking: How to Rate your Company’s Performance against
the World’s Best, Wiley, New York, NY.
Yasin, M.M. (2002), “The theory and practice of benchmarking: then and now”, Benchmarking:
An International Journal, Vol. 9 No. 3, pp. 217-43.
Zairi, M. and Whymark, J. (2000), “The transfer of best practices: how to build a culture of
benchmarking and continuous learning – Part 1”, Benchmarking: An International
Journal, Vol. 7 No. 1, pp. 62-78.

About the author

Paula Kyrö is Professor of Entrepreneurship Education at the University of Tampere, Research
Centre for Vocational Education. She holds the positions of Docent in Entrepreneurship
Education in both the University of Jyväskylä and the University of Helsinki, as well as she is
Docent in Entrepreneurship and Management in the Lappeenranta University of Technology.
Paula Kyrö received her first PhD (Educ.) in Adult Education from the University of Jyväskylä in Action research
1997. She has held a position of Assistant Professor in Education in the University of Helsinki,
Professor in Entrepreneurship in the University of Jyväskylä, School of Business and Economics and networking
and she has worked as a visiting Professor in Entrepreneurship in Jönköping International benchmarking
Business School from January 2001 to June 2002. She has also worked in the SME sector as a
Management Consultant for 20 years in her own company, Oy Startup Ab. Paula Kyrö has
written about 80 publications and articles on entrepreneurship, women entrepreneurship,
entrepreneurship and sustainable development, entrepreneurial education, management 105
consulting and professional services. Paula Kyrö can be contacted at: paula.kyro@d1c.fi

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

BIJ
13,1/2 An expert diagnosis system for
the benchmarking of SMEs’
performance
106
Josée St-Pierre and Sylvain Delisle
Université du Québec à Trois-Rivières, Trois-Rivières, Canada

Abstract
Purpose – The purpose of this paper is to show that benchmarking allows SMEs to improve their
operational performance.
Design/methodology/approach – The paper presents a fully implemented expert diagnostic
system which evaluates on a benchmarking basis the performance of SMEs.
Findings – The research results with hundreds of SMEs show that benchmarking allows them to
improve their operational and financial performance thus confirming the usefulness of benchmarking
for SMEs, especially since traditional performance models for large enterprises do not apply well to
SMEs.
Research limitations/implications – Based on data mining techniques, future work should allow
us to significantly extend our knowledge on SMEs, and further improve our evaluation model of SME
performance.
Practical implications – Practitioners and researchers should pay more attention to benchmarking
as a valuable performance evaluation tool, not only for large businesses, but for SMEs as well.
Originality/value – The paper highlights the development and use of a benchmarking-based
“360-degrees” performance evaluation system for SMEs.
Keywords Benchmarking, Small to medium-sized enterprises, Business performance
Paper type Research paper

Introduction
The importance of evaluating performance is obvious for business enterprises in
general, and for SMEs in particular (Yusof and Aspinwall, 2000). Facing increased
competitive pressure due to globalization, as well as increased quality and service
requirements from their customers (Underdown and Talluri, 2002), small- and
medium-sized manufacturers must increase their productivity and their
competitiveness in order to survive and prosper, even if they do not intend to
become “world class” enterprises. One approach to the evaluation of a SME’s situation
is to compare the firm’s business practices and performance with those of a group of
similar firms, that is, to “benchmark” the organization. As management challenges
have increased in complexity, benchmarking has become a strategic tool for
organizations of all sizes.

Benchmarking: An International
Journal
Vol. 13 No. 1/2, 2006
pp. 106-119
q Emerald Group Publishing Limited
1463-5771
DOI 10.1108/14635770610644619
The authors thank the Groupement des chefs d’entreprise du Québec for their contribution to the
PDG project. The authors are members of the Chaire de recherche du Canada sur la performance
des entreprises and they would like to thank the Canada Research Chairs Program, the Canada
Foundation for Innovation, and Canada Economic Development for their financial support of this
research.
 Whatever the type of benchmarking used, internal, competitive or generic, it is Benchmarking
deemed to be a strategic tool that allows the firm to identify possible sources of of SMEs’
improvement in order to increase its performance and its competitiveness (Cassell et al.,
2001; Haughton et al., 1999). The benefits of benchmarking are being more and more performance
recognized, less so in SMEs, however, where its use still remains relatively sparse
(Prabhu et al., 2000; Ahmed et al., 1996). In this last regard, Monkhouse (1995) states
that smaller firms would have a lot to gain by this activity, notably in breaking the 107
entrepreneurs’ “isolation” by providing them with information on what is being done
by other firms similar to their own. This author notes, however, that it is very
important to first convince entrepreneurs of the relevance and usefulness of such a
management tool, thus justifying the significant amount of time spent by themselves
and their key employees on benchmarking activities. Moreover, as Cassell et al. (2001)
have shown, few entrepreneurs turn to benchmarking, giving as an excuse the lack of
time, resources, and even relevance, whereas those who have used benchmarking
recognize its effectiveness and its usefulness for their organization.
A full benchmarking exercise, such as that developed for large enterprises, does not
suit very well SMEs and their reality. Ribeiro and Cabral (2003), in a recent review of
the literature on that topic, summarize the benchmarking process in these four steps:
(1) planning, i.e. decide what will be part of the benchmarking exercise and who
will contribute to it;
(2) information gathering in order to establish the comparison basis;
(3) analysis of the gaps between the enterprise and its partner(s), on an individual
or collective basis; and
(4) adoption or implementation of changes in the enterprise in order to straighten
out the situation, if need be.

Depending on the available resources, such a full benchmarking exercise can last
between several days up to several weeks and is not appropriate for SMEs. Indeed, the
latter are often considered as unique (Julien, 1998), thus it is difficult to find directly
comparable enterprises, and their management often refuses to share confidential
strategic information due to their vulnerability. SMEs also have little financial and
human resources to dedicate to such an activity, especially if the expected effects are
not instantaneous. For all these reasons, SME owner-managers consider benchmarking
as of “little use” – in particular, see Cassell et al. (2001). In addition, Ribeiro and Cabral
(2003) point out that underestimation of the necessary resources to conduct a full
benchmarking exercise can lead to its failure thus causing unjustified financial losses.
Another “brake” on a benchmarking activity is the selection of the enterprise’s
domains or functions to be evaluated, i.e. benchmarked, as noted in Cassell et al. (2001).
This situation is even more challenging in SMEs since their owners-managers often do
not have the required strategic and global view of their enterprise to conduct a
benchmarking exercise ( Julien, 1998).
In spite of these difficulties, it is possible to develop effective benchmarking tools for
SMEs, provided these tools take into consideration SMEs’ characteristics, thus being
different from simple miniatures of the benchmarking tools developed for large
enterprises. Cassell et al. (2001) acknowledge the remote value, for SMEs, of models
developed for large enterprises and the fact that their practices and performance standards
BIJ do not apply usefully to smaller enterprises. Different strategic goals, a more complex
13,1/2 business environment, and limited resources are all factors contributing to the justification
of SME-specific benchmarking (Hendry et al., 1995; Cassell et al., 2001). Nonetheless, these
differences do not question the importance of central concepts such quality, performance,
or competitiveness; it simply shows that these concepts must be applied in a customized
fashion to the specific needs of SMEs. The very notion of benchmarking must be tailored to
108 SMEs in order to target those that wish to progress, and not only those that wish to become
world leaders or the best in their market since there are relatively few SMEs that set
themselves such goals. In fact, in order to be relevant for SMEs, a benchmarking tool must
remain simple, comprehensive, not too demanding in terms of resources, and it must be
able to guide owners-managers toward action when appropriate.
Our answer to this challenge of the utmost importance for SMEs was to design,
develop and test an advanced computerized benchmarking tool. This tool, the PDG
(performance, development, growth) system, constitutes the foundation of our work on
SMEs benchmarking, a domain in which our lab has been involved for many years. As
mentioned in Denkena et al. (2003), the implication of researchers in such a project is
justified by their interest in a better understanding of SMEs, and the contingency
factors that influence their development, with the ultimate goal of increasing the state
of the art on SMEs and putting forward valid SME-related theories. As suggested in
Cassell et al. (2001) and Voss et al. (1994), rigor put in by researchers in such projects
will contribute to a high level of quality and relevance in the benchmarking tools thus
developed, while allowing a better understanding of the usefulness and effectiveness of
benchmarking in the realm of SMEs. This is what we discuss in this paper, taking as
the basis of our presentation the PDG system.

The PDG benchmarking system: a bird’s eye view

Many authors have criticized performance measure models because they rely on
indicators that:
.
have little predictive power as to the future performance of the enterprise;
.
do not allow the identification of the problems’ sources, whenever there are
problems; and
.
do not provide sufficient granularity to target immediate specific actions (Tan
et al., 2003).

The particular model we have developed (Figure 1) address these shortcomings by

providing a global perspective on the enterprise, by allowing the identification of
relevant improvements based on relationships between the enterprise’s way to do
things (or its implemented management and production practices) and its results, and
by allowing the identification of potential difficulties the enterprise might experience
that could spoil its future performance, i.e. its vulnerability.
The current version of the PDG benchmarking system is the result of a long and
demanding development process, a project that has been going on for more than
six years. Many people were involved, all of them in various specialization fields
(i.e. management, marketing, accounting, finance, human resources, engineering,
innovation, information technology, etc.): researchers, graduate students, research
professionals and, of course, SME entrepreneurs.
 Evaluated Practices Results/Performance Benchmarking
Function of SMEs’
performance
Training;
Voluntary leave rate;
Human Evaluation;
Work relations;
Resources Remuneration;
Participative
Workforc 109
productivity; etc.
management; etc.

Production

Marketing

Control

Figure 1.
Vulnerability Effectiveness The PDG conceptual
<< FUTRE >> <<PAST>> framework

The goal of the PDG system is to evaluate a SME, from an external perspective and on
a comparative basis, in order to produce a diagnosis of its performance and potential,
complemented with relevant recommendations. Right from the beginning, the project
evolved around the concept of a questionnaire that would be used to collect relevant
information items on the SME to be evaluated. This choice allowed us to take into
consideration the availability of resources in SMEs, as well as their need for
confidentiality during data processing. Data extracted from the questionnaire is
computerized and fed into a complex benchmarking system that carries out the
evaluation. The latter is performed in approximately three minutes by contrasting a
particular SME with an appropriate group of SMEs for which we have already
collected relevant data. The PDG’s output, which we call the PDG report, or report in
Figure 2 is a detailed report in which 28 management practices (concerning human

data and results

Entrepreneur Oracle Database
(SME)

Lab Figure 2.
questionnaire,
financial data

The PDG system:

report

questionnaire
financial data

evaluation of SMEs, from

report

an external perspective
report

and on a comparative
PDG Expert information Multidisciplinary basis, in order to produce a
Diagnosis Team of Human diagnosis of their
Intermediary
System expertise Experts performance and potential
Partner
BIJ resources management, production systems and organization, market development
13,1/2 activities, accounting, finance and control tools, effectiveness and vulnerability), 20
results indicators and 22 general information items are evaluated, leading to 14
recommendations on short term actions the evaluated SME could undertake to improve
its overall performance. The PDG system is fully implemented and the actual version
has been in production for two years. So far, we have produced more than 600 reports
110 and accumulated in the PDG’s database the evaluation results of 450 different
manufacturing SMEs.
Figure 2 shows the PDG in a systemic fashion, connected to an Oracle database that
collects all the relevant data for benchmarking purposes. A team of multidisciplinary
human experts constantly monitors the PDG reports in order to ensure the evaluation
is accurate and the recommendations are relevant. This validation phase, which
always takes place before the report is sent out to the evaluated SME, is an opportunity
to make further improvements to the PDG system, whenever appropriate. It is also a
valuable means for the human experts to update their own expertise on SMEs. Figure 2
also shows that an intermediary partner is part of the process in order to guarantee
confidentiality: nobody in our lab knows to what companies the data are associated.
A good deal of multi-domain expertise and informal knowledge engineering was
invested into the design of the PDG’s extended questionnaire, as well as into the
benchmarking system. In fact, at the early stage of the project, it was even hoped that a
traditional (symbolic) expert-system approach would apply naturally to the task we
were facing – a prototype expert system was developed for a subset of the PDG
dealing only with human resources. However, reality turned out to be much more
difficult than anticipated. In particular, the knowledge acquisition and knowledge
modeling phases (Santos et al., 2002; Schreiber et al., 2002; Turban and Aronson, 2001)
were very demanding, especially in the context of a multidisciplinary domain such as
that of SME benchmarking.
The questionnaire’s and the report’s content have been elaborated from the
literature on SME performance and bankruptcy, enriched by SMEs experts who
identified the practices that can have a positive influence on SMEs’ long-term financial
performance, and this for every manufacturing SME business function, and validated
by entrepreneurs. On top of financial indicators that are commonly used for all
enterprises, including SMEs (Cassell et al., 2001; Frigo, 2002), non-financial and mixed
indicators were also added, depending on the management practices evaluated. Our
approach differs notably from that of Denkena et al. (2003) who have identified
statistically-based performance indicators from a database, without any theoretical
validation, and which may lead to an unstable evaluation model in time. A
benchmarking model for SMEs provides a reliable foundation for a benchmarking
system and prevents unjustified modifications due to factors such as the latest
business environment buzz.
The current version of the PDG system, although not implemented with symbolic
artificial intelligence techniques, e.g. knowledge base of rules and facts, inference
engine, etc. qualifies as a hybrid diagnosis-recommendation expert system. The
expertise hidden in the PDG system comes from human experts that have identified the
appropriate contents, selected relevant information, associated adequate relative
weights to evaluation elements, and prepared suitable recommendations for the
evaluated SME. The knowledge, information and algorithms it uses allow it to produce
outputs that only a human expert, or in fact several human experts in different Benchmarking
domains, would be able to produce in terms of diagnosis quality, including of SMEs’
recommendations. Over 90 percent of the reports produced by the PDG system need no
modification whatsoever before being returned to the evaluated SME. These reports performance
contain mostly colored diagrams and simple explanations that are formulated in plain
English (or French) so that SME entrepreneurs can easily understand and interpret
them. 111

A closer look at the PDG’s main components

If we look at the PDG from a systemic viewpoint, as shown in Figure 3, we see that:
(1) the input consists of a broad-coverage questionnaire filled by the SME and its
financial statements, including the accompanying notes, of the last five years;
(2) processing is performed by the expert diagnosis system; and
(3) the output consists of a detailed benchmarking evaluation of the SME’s
performance, plus recommendations.

In this section, we will take a look at these three main components in order to better
understand the PDG system and how it works.

The PDG’s input: the SME’s data

Taking into consideration the availability of resources in SMEs, their reluctance to
verbally share their strategic data, and the fact that data gathering ought to be done in
a relatively short period of time in order to keep the owner-manager’s interest, we have

Figure 3.
A simplified view of the
PDG system and its main
components: from a
broad-coverage
questionnaire and
financial statements to a
detailed benchmarking
evaluation and associated
recommendations
BIJ opted for a questionnaire to collect all the relevant data for benchmarking purposes.
13,1/2 It normally takes about four hours to the owner-manager, usually in collaboration with
members of her or his management team, to fill in the questionnaire.
The questionnaire on which is collected the data from the manufacturing SME to be
evaluated is broad although not deeply specialized in any of the business’ functions.
It consists of 19 pages (plus a one-page appendix of term definitions) organized in these
112 five main sections:
(1) An indication of the SME’s preference as to what reference group it wishes to be
compared to. This information is crucial for benchmarking purposes. It is our
experience that SMEs often have a hard time identifying their reference group,
especially the first time around. This is one area where human expertise is
needed: an expert of our lab must approve the group selection before proceeding
to the evaluation itself. Indeed, the reference group (or group of peers) against
which the SME will be compared should be selected by the entrepreneur
according to her or his own strategic goals. The latter do influence significantly
the enterprise’s structure, its management and production practices, the specific
equipments and tools it uses, and so on. For instance, if the owner-manager
wishes to improve the enterprise’s innovation capabilities, she or he can select a
reference group based on R&D activities; if the goal is rather to expand the
enterprise’s market, then the reference group should be based on sales growth
or export rate. But again, if the goal is instead to measure the enterprise’s
competitiveness, the benchmarking could be made relative to SMEs in the same
branch; if the entrepreneur only wishes to perform a kind of self-evaluation,
then SMEs of the same size (i.e. same number of employees) could constitute the
reference group. This variety of comparison bases is in agreement with SMEs’
heterogeneousness and emphasizes the fact that the SME’s branch of industry
is not always the best criterion for benchmarking purposes, depending on the
enterprise’s development stage and the owner-manager’s development strategy
(Raymond and St-Pierre, 2004). The selection of the most appropriate reference
group plays a key role in benchmarking since, as Sousa et al. (2003) have shown,
it greatly influences the performance indicators that will be used by enterprises
throughout the exercise.
(2) A section to be filled by the business owner-manager: it includes questions
about the owner’s background, the existence of a board of directors, the strategy
regarding new technologies, the strategy on market development, the usual
sources of information on a dozen of subjects, the existence of a designated
person in charge in various services or departments, an opinion on the firm’s
positioning relative to its competitors, a forecast on the firm’s financial
situation, and so on.
(3) A section to be filled by the human resources manager: it includes questions
about the number of employees in various categories, employee training,
employee evaluation, sharing of information in the organization, a forecast on
the evolution of several employee categories, and so on.
(4) A section to be filled by the controller or the general manager: it includes
questions about the enterprise’s ownership, its main products or services, the
geographical sales distribution, collaborations or alliances with other
 organizations, the value of production equipment, research and development, Benchmarking
the importance of sales on credit, dependence on principal clients and suppliers, of SMEs’
accounting and financial tools, information technologies, credit, interest rate,
and so on. performance
(5) A section to be filled by the production manager: it includes questions about the
nature of the enterprise’s manufacturing operations, the usage of equipment
over the year, subcontracting, certification, standards, continuous improvement 113
activities, quality, safety, use and proficiency in the use of various equipments
and technologies, integration of production systems, control of operating costs,
and so on.

Once all the data on a questionnaire, plus the complete financial statements of the last
five years, have been manually verified, they are typed in via a software dedicated to
this task and will finally end up in the PDG’s database, ready for benchmarking. It is
worth mentioning that the questionnaire’s data complexity and diversity (i.e.
qualitative, quantitative, historical, conjectural, perceptual, etc.) must be matched by
the expert diagnosis processor that must contain all the required information and
encoded knowledge to correctly perform the evaluation.

The PDG’s central processing: the expert diagnostic processor

The heart of the expert diagnosis-recommendation system can best be described in
relation to the report generator component, which acts as the main program. The eight
main steps of processing are:
(1) The reference group is created according to a number of criteria, occasionally
after several iterations. Indeed, it is sometimes necessary to create a few
tentative reference groups in order to select the most appropriate one, thus
ensuring the statistical validity of the benchmarking results, while preserving
confidentiality of all SMEs. This processing takes place within the Oracle
database.
(2) A number of statistical computations (averages, medians, distributions) related
to the reference group are performed within the SAS statistical package, after
data from the Oracle database has been fetched.
Starting from here, the remaining processing is essentially performed by a Microsoft
Excel application:
(3) Statistical data is passed from SAS to Excel via an open database connectivity
(ODBC) connection.
(4) A bunch of automatic, numerical Excel computations take place at this stage in
order to define the relative positioning of the evaluated SME to its reference
group. This relative positioning will then be used to define the appropriate
graphical representations (including colors), as well as related comments and
recommendations.
(5) Graphics and various representations associated with the numerical values just
computed are then produced.
(6) Textual comments on practices and results are computed, in correlation with
the computations completed in the previous steps.
BIJ (7) An evaluation summary, to appear on the report’s first page, is produced. For
13,1/2 each of the seven main business functions, it presents a synoptic bar diagram
showing the evaluated SME’s relative position, plus two recommendations as to
what actions should be considered in the next few years.
(8) The actual PDG output report, containing the benchmarking results, is then
produced and saved on disk. The report is displayed on the screen by the Excel
114 application.

Step (6) deserves further explanations. In fact, why bother with textual comments to be
added to the already abundant information present in the report? The main goal of the
PDG report is to guide entrepreneurs in the identification of potential sources of
improvement in their business’ performance. It turns out that the average entrepreneur
cannot easily grasp all the information involved in such a report based solely on
graphical representations (e.g. charts). This information is thus analyzed by the PDG
system and textual comments on practices and results are automatically produced to
facilitate the entrepreneur’s interpretation task when reading the report.
The actual mechanisms used to automatically produce these textual comments are a
bit tricky. In brief, they are similar to early artificial intelligence techniques devised for
natural language processing based on templates. A prototypical comment contains
fixed and variable segments. The latter are determined according to the specific
numerical values computed for the evaluated SME (during previous steps).

The PDG’s output: benchmarking results and recommendations

The diagnosis produced by the PDG system through benchmarking is presented in an
easy-to-read report that uses mostly graphics, colored (colors translates into shades of
gray in Figure 4) diagrams and charts, and numerical results – typical SME
entrepreneurs are not fond of long textual reports.
The report consists of ten pages. The first page presents a summary of the
evaluated SME along the main enterprise functions, using a cursor-based graphical
representation. The report’s next two pages present a summary of the SME’s reference
group characteristics, all illustrated with colored pie charts and bar charts. This
presentation of the reference group is much needed since entrepreneurs tend to think
that their enterprise is unique and thus incomparable; it thus contributes to the
justification of the benchmarking approach by showing that relevant comparables do
exist. Then a one-page summary on each of the following topic is presented: human
resources, production systems, production management, sales/marketing,
management and control, effectiveness, and vulnerability.
As we evoked before, one of the criticisms about performance measure systems and
benchmarking tools is the difficulty, even the impossibility, to establish a connection
between results and potential actions the enterprise should undertake to improve its
situation. The PDG report, which is targeted especially at SME owners-managers, does
contain explicit recommendations pertaining to every business function and, whenever
appropriate, specific suggestions are made as to how current practices could be
modified to improve the enterprise’s performance. Thus, the relationships between
results, practices, and action, are explicitly handled by the PDG’s
diagnosis-recommendation expert system.
 Benchmarking
of SMEs’
performance

115

Figure 4.
A typical page of a PDG
report contains an
evaluation of a SME’s
management practices
(section A above: which
generate four to six results
for each main business
function), and results
(section B above: two to
five results, depending on
the business function, and
relative to its reference
group), plus comments
and recommendations
(section C above: which
are formulated relative to
their situation compared
to the reference group)

On each of these pages appear brief textual comments and recommendations, when
appropriate, on the SME’s practices and results (in section C, in Figure 4). Here is an
example comment on practice evaluation of human resources (the variable segment is
in italics):
Your human resources management practices are generally more advanced than those of
your reference group. You could improve your performance, in particular by
implementing participative management to involve employees in the growth of the business.
And here is an example comment on human resources results (the variable segment is
in italics):
Your overall effectiveness at managing human resources is comparable to that of your
reference group. You should pay attention to why certain managerial jobs have a high rate of
voluntary departure, with the objective of lowering hiring and training costs.
One final remark on the PDG report. Before it is sent to the evaluated SME, at least one
expert from our lab validates its contents. This expert must complete the comments
appearing on the first page by selecting the most appropriate comments from a
BIJ precompiled list of potentially relevant candidates. These textual comments must be
13,1/2 determined from a global perspective in which the human expert will consider
interrelationships between various aspects that appear independently in the report.

Evaluating the usefulness of the PDG system

116 An experimental evaluation was performed by St-Pierre et al. (2002) on the inventory of
reports produced to that date for 307 clients, 258 first-time and 49 second-time users.
The authors hypothesized that the use of benchmarking would have a positive effect
on enterprise operational and financial outcomes, and tested this by comparing the
results involved for the first- and second-time populations as reported in their profiles.
Although not all differences are significant, nonparametric statistical analysis (t- and
z-scores) showed general support for the hypothesis. Further examination of the
analysis indicated that the 49 second-time users “showed marked improvement in their
operational performance from the first to second year”. An attempt was then made to
gain insight into the relationships between the factors in play using structural equation
modeling (partial-least-squares). It suggested that benchmarking has a positive effect
on operational performance (g ¼ 0.13) and financial performance (g ¼ 0.08), and tends
to lead to the implementation of best practices (g ¼ 0.16). The authors explain the
lower gamma for financial performance as reflecting the time required for increases in
productivity to show up on the bottom line.
How many SMEs have used our benchmarking tool, once, twice or more times? The
number of first time users and the number of second time or more, i.e. repeat users,
provides another indication of the quality and relevance of the PDG system and, more
specifically, of the benchmarking reports which the system produces. Throughout the
last four years, 459 SMEs have used the PDG system more than 600 times, with 159
using the benchmarking system twice or more. We believe these numbers represent a
relatively satisfactory participation rate, given the constraints mentioned above and
considering the fact that benchmarking itself is a relatively new management practice
which was relatively unknown even two years ago, especially by small enterprises.
The particular lab in which the PDG project was undertaken is mainly concerned
with the development of scientific expertise on the study and modeling of SMEs’
performance, including a variety of interrelated subjects such as finance, management,
information systems, production, technology, etc. Thanks to these hundreds of user
evaluations we have been able to assemble an important database on manufacturing
SMEs, one which is useful in scientific research on small- and medium-sized
enterprises. Indeed, researchers at our lab and institute have already capitalized on this
multidimensional database to reveal, amongst other things: the determining effect of
the entrepreneur’s characteristics on the development of her or his business (Raymond
and St-Pierre, 2003); that R&D activities in SMEs are influenced by subcontracting
relationships maintained by a SME with prime contractors (Raymond and St-Pierre,
2004); that the innovation process in SMEs is done according to particular strategic
goals (St-Pierre and Mathieu, 2003); that an enterprise’s commercial dependency plays
an important role in the presence (or absence) of certain management practices, as well
as on its performance (Raymond and St-Pierre, 2002). On top of the actual
benchmarking reports produced for SMEs, these scientific results are another very
important indirect output that clearly underlines the usefulness of the PDG system.
Conclusion Benchmarking
Although benchmarking has proven to be valuable for large businesses and of SMEs’
organizations for quite some time, serious doubts existed as to its usefulness for
smaller businesses until recently. We have presented a fully implemented expert performance
diagnostic system which evaluates on a benchmarking basis the performance of SMEs.
The PDG system has been in use for several years and has gone through a constant
and quite challenging evolution in order to meet both the needs of SME-oriented 117
research and the production of benchmarking reports for hundreds of SMEs. The
current version of the PDG ended up as knowledge-packed system built on database
technology that allows it to produce outputs that only a human expert, or in fact
several human experts in different domains, would be able to produce in terms of
diagnosis and recommendation quality. The output report contains mostly colored
diagrams and simple explanations that are formulated in plain English (or French) so
that SMEs entrepreneurs can easily understand it.
Our research results show that benchmarking allows SMEs to improve their
operational and financial performance thus confirming the usefulness of
benchmarking, especially since traditional performance models for large enterprises
do not apply well to SMEs. These results also confirm the value of the
recommendations included in the PDG report concerning short-term actions to be
undertaken to modify management practices. We think the PDG system proves that, if
the benchmarking approach is tailored to SMEs’ characteristics, an adequate tool can
be devised and used to help them improve their performance. However, our experience
shows that this is no easy endeavor. SMEs are a multifaceted reality to which
established expert-system and decision-support-system methodologies cannot be made
to apply without substantial adjustments.
The main focus of our current work is on the development of data warehouses and
data mining algorithms to facilitate statistical processing of data and extend
knowledge extraction capabilities. Indeed, we are about to activate a new data
warehouse containing lots of data on SMEs and start our first experimentations with
data mining techniques. Data warehousing has many practical uses in our
SME-oriented context and it will, along with data mining techniques, positively
affect the way researchers use the rich data we have collected and continue to collect on
SMEs. We hope to significantly extend our knowledge on SMEs, and further improve
our evaluation model of SME performance. For instance, a better understanding of the
way SMEs adopt and put to use best business practices and the influence these have on
the enterprise’s performance will allow us to ensure that the PDG system remains
relevant and in tune with SMEs’ specific context and needs. Other goals, amongst
many, are: to get a better understanding of SME performance, whether defined from
financial data or not; to master the implementation process of certain business
practices and the required period of time before any positive effect on the enterprise’s
performance can be detected; evaluate to what extend the environment influences the
development of SMEs; and so on.

References
Ahmed, N.U., Montagno, R.V. and Firenze, R.J. (1996), “Operations strategy and organizational
performance: an empirical study”, International Journal of Operations & Production
Management, Vol. 16 No. 5, pp. 41-53.
BIJ Cassell, C., Nadin, S. and Gray, M.O. (2001), “The use and effectiveness of benchmarking in
SMEs”, Benchmarking: An International Journal, Vol. 8 No. 3, pp. 212-22.
13,1/2
Denkena, B., Apitz, R. and Liedtke, C. (2003), “Knowledge-based benchmarking of production
performance”, paper presented at Business Excellence ’03, 1st International Conference on
Performance Measures, Benchmarking and Best Practices in New Economy, Guimaraes,
10-13 June.
118 Frigo, M.L. (2002), “Nonfinancial performance measures and strategy execution”, Strategic
Finance, Vol. 84 No. 2, pp. 6-9.
Haughton, M., Grenoble, W., Thomchick, E. and Young, R. (1999), “The role of benchmarking in
the performance of the import process”, International Journal of Physical Distribution &
Logistics Management, Vol. 29 No. 9, pp. 511-68.
Hendry, C., Arthur, M.B. and Jones, A.M. (1995), Strategy Through People: Adaptation and
Learning in the Small-Medium Enterprise, Routhledge, London.
Julien, P-A. (1998), The State of the Art in Small Business and Entrepreneurship, Ashgate,
Aldershot, p. 479.
Monkhouse, E. (1995), “The role of competitive benchmarking in small to medium sized
enterprises”, Benchmarking, Vol. 2 No. 4, pp. 41-50.
Prabhu, V., Yarrow, D. and Gordon-Hart, G. (2000), “Best practice and performance within
northeast manufacturing”, Total Quality Management, Vol. 11 No. 1, pp. 113-22.
Raymond, L. and St-Pierre, J. (2002), “Performance effects of commercial dependency for
manufacturing SMEs”, in Moore, D.L. and Fullerton, S. (Eds), International Business
Trends: Contemporary Business Readings, National Meeting of the Academy of Business
Administration, Key West, FL.
Raymond, L. and St-Pierre, J. (2003), “Entrepreneurial antecedents and performance outcomes of
organisational development in manufacturing SMEs”, paper presented at 6th International
Conference on Quality and Management for Organisational Development, Paris, October.
Raymond, L. and St-Pierre, J. (2004), “Customer dependency in manufacturing SMEs:
implications for R&D and performance”, Journal of Small Business & Enterprise
Development, Vol. 1 No. 1, pp. 23-33.
Ribeiro, L.M.M. and Cabral, J.A.S. (2003), “A benchmarking methodology for the metalcasting
industry”, paper presented at Business Excellence ’03, 1st International Conferences on
Performance Measures, Benchmarking and Best Practices in New Economy, Guimaraes,
10-13 June.
Santos, J., Vale, Z. and Ramos, C. (2002), “On the verification of an expert system: practical
issues”, Lecture Notes in Artificial Intelligence #2358, Springer, New York, NY, pp. 414-24.
Schreiber, G., Akkermans, H., Anjewierden, A., de Hoog, R., Shadbolt, N., Van de velde, W. and
Wielinga, B. (2002), Knowledge Engineering and Management: The Common KADS
Methodology, MIT Press, Cambridge, MA.
Sousa, S.D., Aspinwall, E.M. and Rodrigues, A.G. (2003), “Performance measures in English
small and medium enterprises: survey results”, paper presented at: Business Excellence
’03, 1st International Conference on Performance Measures, Benchmarking and Best
Practices in New Economy, Guimaraes, 10-13 June.
St-Pierre, J., Raymond, L. and Andriambeloson, E. (2002), “Performance effects of the adoption of
benchmarking and best practices in manufacturing SMEs”, paper presented at Small
Business and Enterprise Development Conference, The University of Nottingham,
Nottingham.
St-Pierre, J. and Mathieu, C. (2003), “Innovation in Canadian SMEs: the process, characteristics of Benchmarking
firms and their environment”, paper presented at International Council for Small Business,
48th Conference, Belfast, 15-18 June. of SMEs’
Tan, K.H., Platts, K. and Noble, J. (2003), “Building performance through in-process performance
measurement: toward an ‘indicative’ scorecard for business excellence”, paper presented
at Business Excellence ’03, 1st International Conference on Performance Measures,
Benchmarking and Best Practices in New Economy, Guimaraes, 10-13 June.
119
Turban, E. and Aronson, J.E. (2001), Decision Support Systems and Intelligent Systems,
Prentice-Hall, Englewood Cliffs, NJ.
Underdown, R. and Tallury, S. (2002), “Cycle of success: a strategy for becoming agile through
benchmarking”, Benchmarking: An International Journal, Vol. 9 No. 3, pp. 278-92.
Voss, C.A., Chisea, V. and Coughlan, P. (1994), “Developing benchmarking and self assessment
frameworks in manufacturing”, International Journal of Operations & Production
Management, Vol. 14 No. 3, pp. 83-97.
Yusof, S.M. and Aspinwall, E.M. (2000), “Critical success factors in small and medium
enterprises: survey results”, Total Quality Management & Business Excellence, Vol. 11
Nos 4-6, pp. S448-62.

About the authors

Josée St-Pierre holds a PhD in finance from Laval University (1990) and is a full Professor of
Finance in the Department of Management Science at the University of Quebec, in Trois-Rivières
(UQTR). She is Director of the J. Armand Bombardier Chair for research on interenterprise
relations and risk management and director of the laboratory for research on business
performance at the research institute for SMEs at l’UQTR. Her current research interests focus
on SMEs performance, innovation, financial problems, risk management, benchmarking
and valuation. Josée St-Pierre is the corresponding author and can be contacted at: josee.st-
pierre@dr.cgocable.ca
Sylvain Delisle holds a PhD in Computer Science from the University of Ottawa (1994). He is a
full Professor in the Department of Mathematics and Informatics at the University of Quebec, in
Trois-Rivières (UQTR). His research interests include artificial intelligence, databases and data
mining, computer-based diagnosis systems for SMEs, and electronic business. Delisle is an
active member of the research institute for SMEs at l’UQTR. He also is a member of the Canada
Research Chair on Enterprises Performance. His research is funded by various agencies,
including an individual grant from the National Science and Engineering Research Council of
Canada (NSERC).

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

BIJ
13,1/2 Performance measures in English
small and medium enterprises:
survey results
120
Sérgio D. Sousa
Manufacturing and Mechanical Department, School of Engineering,
University of Birmingham, Birmingham, UK
School of Engineering, University of Minho, Braga, Portugal
Elaine M. Aspinwall
Manufacturing and Mechanical Department, School of Engineering,
University of Birmingham, Birmingham, UK, and
A. Guimarães Rodrigues
School of Engineering, University of Minho, Braga, Portugal

Abstract
Purpose – To determine the current state of knowledge related to performance measures and their
degree of implementation in small and medium enterprises (SMEs) in England.
Design/methodology/approach – The paper starts with a literature review and then goes on to
discuss the methodology used. The survey is briefly presented together with the analysis of the
resultant data. General opinions regarding performance measurement in English SMEs are described,
including the most important measures and the biggest obstacles to the adoption of new ones.
Hypotheses about differences between groups are tested and discussed.
Findings – This work concludes that there is a gap between the theory/knowledge of performance
measures and the practice in English SMEs. Training of employees and difficulty in defining new
performance measures were highlighted as the major obstacles to the adoption of new performance
measures.
Research limitations/implications – The low response rate of the survey precludes the
generalisation of the findings.
Practical implications – Innovation and learning measures should be applied more widely.
Originality/value – This paper is relevant to academics and SME managers because it supports the
existence of a gap between the theory of performance measurement and its degree of implementation.
In addition, it introduces both theoretical information on performance measurement, including that
based on the balanced scorecard perspectives, and practical information from a survey conducted in
English SMEs.
Keywords Performance measures, Small to medium-sized enterprises, Balanced scorecard,
Total quality management, England
Paper type Research paper

Benchmarking: An International
Journal
Vol. 13 No. 1/2, 2006
pp. 120-134
q Emerald Group Publishing Limited
1463-5771
DOI 10.1108/14635770610644628
The authors are grateful to all the companies that participated in this survey. This work was
partially supported by British Chevening Scholarships (Grant POR 0100109) and Fundação para
a Ciência e a Tecnologia (Grant SFRH/BD/6939/2001). This paper is an extended version of the
work presented at the First International Conference on Performance Measures, Benchmarking
and Best Practices in New Economy – Business Excellence 2003, 10-13 June 2003, University of
Minho, Portugal.
Literature review Performance
For the purpose of this research “performance measurement” (Neely et al., 1995) has measures in
been defined as the process of quantifying the efficiency and effectiveness of action,
and “performance measure” as a metric used to quantify that action. Small and English SMEs
medium enterprises (SMEs) were taken to be those companies with less than 250 (50 for
small ones) workers (Commission of the European Communities, 2003) and:
.
no more than 25 per cent of the capital or voting rights were held by one or more 121
enterprises which were not, themselves, SMEs; and
.
the annual turnover was less than e40 m (e7 m for small companies) or the total
balance sheet was less than e27 m (e5 m for small companies).

Traditional methods of measuring a company’s performance by financial indices alone

have virtually disappeared from large organisations (Basu, 2001). Non-financial
measures are at the heart of describing strategy and of developing a unique set of
performance measures that clearly communicate strategy (Kaplan and Norton, 1992,
1996), and help in its execution (Frigo, 2002).
Frigo (2002) reported the existence of a gap between strategy and performance
measures, which failed to support the communication of strategy within an
organisation. André and Saraiva (2000) noted that there was quite a large gap
between available models and current company practices in Portuguese companies.
Hudson et al. (2001) concluded that although there was a widespread acceptance of
the value of strategic performance measurement amongst the SMEs that they
studied, none had taken steps to redesign or update their current performance
measurement systems.
Many excellence models and performance measurement frameworks, like the
EFQM (2001) excellence model, Kanji’s business scorecard (Kanji and Sá, 2002), the
performance prism (Neely et al., 2002), and the balanced scorecard (Kaplan and Norton,
1992), have proposed ways of using the TQM philosophy. According to Ahmed (2002),
the most popular ones to have drawn the attention of researchers include the balanced
scorecard and the EFQM. Kanji and Sá (2002) state, for example, that the new approach
to performance measurement suggested in the balanced scorecard is consistent with
business excellence and TQM. The balanced scorecard is relevant to both small and
large organisations, however, neither a comprehensive literature review nor any
empirical research exists on implementing the balanced scorecard in SMEs (Andersen
et al., 2001).
The interest, over the last decade, in TQM and quality awards has highlighted the
importance of performance indicators in achieving quality excellence. Quality
measures represent the most positive step taken to date in broadening the basis of
business performance measurement (Bogan and English, 1994). Models of excellence
and improvement initiatives based on TQM principles reflect the importance of not
only complying to specifications but also to delighting an organisations’ stakeholders.
The relationship between TQM practice and organisational performance is
significant (Samson and Terziovski, 1999), and TQM implementation correlates with
quality performance (Brah et al., 2002), despite some contradictory cases (Shaffer and
Thomson, 1992; Ittner and Larcker, 1997; Sterman et al., 1997; Wilbur, 2002). Many of
the failures of TQM in small organisations are related to bad implementation strategies
and processes (Hansson and Klefsjo, 2003). Wood and Childe (2003) showed that it was
BIJ possible to establish relationships between process improvement actions and
13,1/2 performance requirements.
The adoption of the process approach to quality management systems (QMS) was
one of the most important aspects of the year 2000 revisions of ISO 9001 and ISO 9004
(Hooper, 2001). The new ISO 9001 standard (ISO, 2002) requires fact-based decisions
and continual measurement and improvement of performance results (Karapetrovic
122 and Willborn, 2002). These changes have narrowed the gap between the requirements
of a QMS and those of the EFQM excellence model. Both reinforce the need to measure
not only the critical success factors of an organisation but also the satisfaction of its
stakeholders, to allow and assure continuous improvement aligned with strategy.
Juran and Godfrey (1999) and Campanela (1999) considered quality costs to be the
main driver, when selecting quality improvement projects. This can also be done with
the support of the balanced scorecard, making it a strategic management tool as
suggested by Cobbold and Lawrie (2002).
The EFQM (2003) recognises that organisations, on their journey to excellence, may
show different levels of maturity. The selection of the best approach to measure the
effectiveness of a system will ultimately be based on the maturity of the quality efforts,
the type of organisation or process, and other TQM tools applied concurrently
(Campanela, 1999). Brah et al. (2002) reported that the size of a company and the extent
of its experience with TQM affect the rigor of implementation and the resulting level of
performance quality. However, the nature of a company (manufacturing or service)
does not seem to have a significant effect on either aspect.
Hudson et al. (2001) concluded that a discrepancy between theory and practice was
identified in the development processes employed by SMEs, including a lack of
strategic forethought, lack of communication between managers and the lack of a
structured process for development. They also suggest that there are substantial
barriers to strategic PM systems’ development in SMEs. Neely et al. (1995) pointed out
that measurement is a luxury for SMEs – success and failure are obvious. They have
concluded that the cost of measurement is an issue of great concern to managers in
SMEs.

Methodology
The steps followed in this research are similar to those followed by Saraph et al. (1989)
and Yusof and Aspinwall (2000b).
Following a literature review, the subject of performance measurement was
discussed with both academic and non-academic specialists and hypotheses were
formulated. This provided the basis for the construction of a questionnaire which was
pre-tested and revised. The final survey form was sent by e-mail, to privately owned
SMEs in England (both from the service and industrial sectors).
The data was analysed using the SPSS package v11.0. The reliability and validity of
the questionnaire were also verified. A test for possible bias from respondents was
analysed as suggested by Armstrong and Overton (1977).

Survey
The questionnaire consisted of three main sections: the company background, the level
of knowledge about performance measures, and the use of specific performance
measures. The first section was intended to determine general information like number
of employees, sector of activity, number of clients, types of product made, whether a Performance
certified quality system was held, the level of TQM and quality measures adoption and measures in
confirmation that the company was indeed an SME. Each respondent was also asked to
select, from a list of nine, the quality initiatives that had been adopted in their English SMEs
company. In addition, they were asked to state their company’s strategic objectives to
establish whether or not they adopt adequate performance measures to track their
evolution. 123
The second section consisted of 22 statements about the performance measurement
system of the company, including aspects such as the company’s strategy, the selection
of performance measures, their implementation and the results. The respondents were
asked to rate their degree of agreement with each statement according to a five-point
Likert scale from 1 “strongly disagree” to 5 “strongly agree”. Zero was added in case of
doubt. This section also contained a question to determine the most important
performance measures used in the company, and one for the obstacles likely to be
encountered if adopting new ones. The actual criteria that allow companies to win new
orders, as suggested by Neely et al. (1994), were also assessed.
The balanced scorecard (Kaplan and Norton, 1992, 1993) was chosen as the basis for
the third section of the questionnaire mainly because of its simplicity, general
acceptance among practitioners and researchers, and its close association with
strategy (Kaplan and Norton, 1996). The objective of this section was to investigate the
importance and use of different performance measures. A Likert scale similar to that
used in the second section was used to rate the importance and the use of each measure.

Questionnaire reliability and validity

The reliability of the questionnaire, which measures internal consistency, was studied
through Cronbach’s a. This method allows for the calculation of the a coefficient if one
variable is removed from the original set, making it possible to identify the subset that
has the highest reliability coefficient. If all the results are above 0.7, the scales are
judged to be reliable.
In the second section, of the questionnaire, all four groups (components) were
considered reliable after deleting 2 of the 22 statements (variables). The a coefficients
varied between 0.744 and 0.890.
Measures in third section were organised as suggested in the balanced scorecard,
and as can be seen in Table I, all groups of measures were considered reliable.
Within the customer measures group, delivery was not considered reliable, and
therefore, was removed from further analysis. This is not critical to this study because
other components regarding customer performance measures are being considered.
Content validity is always subjectively evaluated by the researcher (Churchil, 1979;
Saraph et al., 1989). An instrument has content validity if it contains a representative
collection of items and if sensible methods of test construction were used (Yusof and
Aspinwall, 2000b). It is strongly believed that the second and third sections of this
survey instrument have content validity as they were well received by the pilot
respondents and by several academics and company managers who assessed them.
Construct validity was tested for the second and third sections using principal
components analysis. Each measure or variable within a component should have a
significant correlation with variables of the same component and low correlation with
others (Hair et al., 1998). The objective of construct validity analysis is to verify if all
BIJ
Balanced scorecard Original number of Number of variables deleted
13,1/2 perspective Measures variables to maximise a a

Internal business Productivity 10 2 0.8180

process
Quality 10 1 0.8466
124 performance
Financial Financial 8 3 0.8710
Innovation and Innovation 8 4 0.8865
learning
Employee 5 0 0.8265
learning
Customer Performance 5 2 0.9095
Table I. Delivery 3 3 Removed
Reliability of measures in Requirements 7 1 0.8370
the third section Satisfaction 6 0 0.7079

the statements that translate the concept under study are unifactorial. If this happens
the group is considered homogeneous.
In the second section, only one variable was deleted to assure that all groups were
unifactorial (Table II), i.e. in each group only one component was extracted, thus all groups
were considered homogeneous. The Kaiser-Meyer-Olkin (KMO) indicator, which is a
measure of sampling adequacy and should not be lower than 0.5, was also verified in all
cases. Variables within each component gave correlations higher than 0.635 in all cases.
Eight variables out of 61 were deleted in the third section to make each group
unifactorial (Table III). The results indicate that in both sections each set of variables
constitutes a homogeneous group. Thus each one translates one concept.

Measures Number of deleted variables KMO Correlations with component 1

Table II. Strategy 0 0.825 0.694-0.833

Principal component Selection of measures 0 0.760 0.720-0.836
analysis of the second Implementation 0 0.741 0.724-0.878
section Results 1 0.687 0.635-0.846

Measures Number of deleted variables KMO Correlations with component 1

Productivity 2 0.653 0.633-0.852

Quality performance 2 0.781 0.509-0.857
Financial 0 0.767 0.725-0.927
Innovation 0 0.784 0.819-0.905
Table III. Employee learning 0 0.778 0.658-0.853
Principal component Performance 0 0.500 0.959-0.959
analysis of the third Requirements 2 0.718 0.628-0.893
section Satisfaction 2 0.645 0.668-0.853
Predictive or criterion-related validity was tested as suggested by Owlia and Performance
Aspinwall (1998) and Yusof and Aspinwall (2000a). A greater use of performance measures in
measures should correspond to a greater understanding of the company’s performance
measurement system. English SMEs
A linear regression analysis was performed on the overall use of performance
measures (from the second section) against the components identified in the third
section. The adjusted R 2 value was 68.2 per cent, suggesting a good fit. To improve 125
this value, a reduction in the number of factors was considered. Using the stepwise
method to select the variables to be added to, or removed from, the regression model,
the adjusted R 2 value increased marginally to 68.6 per cent. The overall perception of
the performance measurement system (OPPMS) can be expressed through the
following model:

OPPMS ¼ 20:19 þ 0:610 * Financial þ 0:433 * Quality_performance

þ 0:418 * Training_of_employees 2 0:326 * Customer_performance

2 0:265 * Innovation

A residual analysis was carried out to validate the assumptions of normality, constant
variance and zero mean.
The model suggests that English SMEs report a higher use of performance
measures if they use financial, quality performance and training of employees’
measures. The negative relationship associated with the use of customer performance
and innovation measures, suggests that these measures may not be perceived as
performance measures.
The results, overall, show that the instrument reflects predictive validity.

Results
The questionnaire was sent to 400 companies and 52 were returned completed. Four of
the respondents were not classified as SMEs resulting in a response rate of 12 per cent.
This is low for a postal survey and so caution must be exercised when generalising
conclusions. The returns were organised into two groups to test possible bias of
respondents. No bias was found and so it can be assumed that non-respondents would
have similar characteristics to the respondents.
Figure 1 shows the breakdown of respondent companies by number of employees.
The wide range of activities covered by respondent companies is shown in Figure 2,
and includes SMEs from the service sector.
The majority of respondents were certified to ISO standards (Figure 3), but only 14
per cent had completed the transition to ISO 9001:2000.
Less than 10
6%

51-250 10-50 Figure 1.

52% 42%
Number of workers
BIJ services
13,1/2 13%
others
logistics
28%
4%

construction
126 8%

metal furniture
pressed 6%
parts
11% electric/
electronic
automotive
plastic/rubber parts
Figure 2. 8%
Sectors of activity components 11%
11%

ISO
None 9000:1994
33% 46%

Others
Figure 3. ISO
6%
SMEs’ quality assurance
9001:2000
system
14%

Continuous improvement or total quality management can be implemented following a

Plan-Do-Check-Act (PDCA) cycle. Thus it is fundamental in the planning phase to
define activities to improve strategic objectives, which will then be monitored.
Respondents selected profitability (53 per cent) as the main strategic objective followed
by quality (22 per cent) and flexibility (10 per cent). When asked about the criteria that
most helped their companies to win orders, manufacturing quality came in first
followed by price (Table IV). It appears that despite other important factors, the

Criteria to win orders Per cent

Manufacturing quality 31
Price 24
Product reliability 10
Wide product range 8
Ability to manufacture customer specials 8
On-time delivery 6
Table IV. Delivery lead time 6
Criteria to win new orders Other 6
quality/price relationship is still of major importance for English SMEs and cannot be Performance
forgotten when initiatives are deployed within an organisation. measures in
Table V presents the quality initiatives already implemented in the respondents’
companies. Setting up a quality department can be explained as a result of ISO English SMEs
standards or simply as a means of implementing the necessary activities to improve
quality and to track their evolution. As employee involvement to improve quality and
establishing measures of quality progress received 65 and 46 per cent, respectively, it is 127
expected that approximately half of companies use measures to assess quality
progress. The same data allow us to conclude that statistical process control, an
efficient tool to understand the variation of a process is used only in 23 per cent of
companies.
General opinions about performance measurement were asked on strategy, selection
of measures, implementation and results (Figure 4), as all of these are important in the
process of continuous improvement. An ANOVA test on the four means showed a
significant difference between them at the 5 per cent level. The assumption of
homogeneity of variances was verified through Levene’s test. The results group has

Quality initiatives Per cent

Employee involvement to improve quality 65

Setting up a quality department 63
Customer satisfaction initiatives 58
Establishing measures of quality progress 46
Business process improvement 35
Developing strategies for total quality 35 Table V.
Supplier involvement programme 29 Quality initiatives
Applying statistical process control 23 adopted by English
Cultural change programme 13 SMEs

3.5

2.5

1.5

0.5 Figure 4.
Overall perception of the
0
performance measurement
Strategy Selection of Implementation Results
system
measures
BIJ the lowest score, meaning that the consequence of using performance measures is not
13,1/2 well understood, and a balance amongst these groups should result in better
performance measurement systems.
Obstacles to the adoption of new performance measures in SMEs include computer
systems issues, lack of top management commitment and the existing accounting
system (Bourne et al., 2000; Neely et al., 1997). The respondents considered (Figure 5)
128 training of employees as the most important obstacle, followed by difficulty in defining
new measures, which could be the result of lack of skills of employees and leadership,
confirming the importance of top management commitment. The cost of the
performance system must always be analysed and is considered of great concern to
SMEs.
According to the literature, companies should adopt a balanced use of the four
groups of measures, as organised in the balanced scorecard. However, respondents
considered some measures more important than others, as shown in Figure 6.
It is curious to note that on-time delivery is not perceived to be a relevant criterion to
win new orders (Table IV) but it is considered the most important performance
measure. This may be because, if a problem occurs in the process or with the supplier it
will be reflected in this measure. In-process quality was perceived to be the second
most important measure.

Balanced scorecard
Grouping all the performance measures together, importance was rated by the
respondents as 3.55, on average, and use as 3.18. This implies that although the
respondents considered performance measures important, they are not used
accordingly. After verifying the homogeneity of the variances, an ANOVA was
performed. This resulted in a p (or significance) value of 6.3 per cent, which, being just
larger than 5 per cent, was too large to be able to conclude a real difference. However,
looking at the four groups separately, financial measures are considered the most
important and are widely used, while innovation and learning measures are rated less
important and are less used (Figure 7).

35
very important
30
important
25
a bit important
20

15
10

5
0
Figure 5. Training of Difficulty Information Cost Leadership Flexibility
Obstacles to the adoption
employees defining system of present
of new performance
measures new currently quality
measures used system
 35 very important Performance
important
measures in
30
a bit important
English SMEs
25

20
129
15

10

ity
y

i ty

et

al
er

st

nt
tim
dg
al

qu
co

ra
liv

qu

bu

ad

ar
de

rts
rw
s

io

le
s
es
e

pa
su
ct
tim

de
oc

du

er
er

g
un
n-

in
iv
pr

tv
ro

Figure 6.
el

m
O

tp

os
In

re
D

co
ni

ilu
C

In Most important
U

fa

performance measures
d
el
Fi

4
3.5
3
2.5
2
1.5
1
0.5
0
Internal FinancialI Innovation Customer Figure 7.
Business and Importance Importance and use of the
Process Learning Use
balanced scorecard

The four groups of measures analysed in this study were assessed to find out if there
was a gap between the perceived importance and the practice or use for each group.
Tests were performed, using the ANOVA with a 5 per cent significance level, to see if
there were any differences between the means of:
.
importance and use of each group of measures;
.
use for companies from the service and industrial sectors;
.
use for SMEs; and
.
use for companies certified according to a quality standard and others.
BIJ The group internal business process exhibited a significant difference between the
13,1/2 importance and use of productivity measures, thus there are measures in this group
that should be put to more use, such as “output per employee or per labour-hour”, “time
spent on each stage of product development”, “time to process an operation”, “number
of errors per unit”, “number of billing errors per unit”, “production volume”,
“absenteeism”, and “injury lost days”. There was insufficient evidence to conclude
130 differences between the importance and use of quality performance measures, meaning
that if they are considered to be important they are being used. The same was also true
for the financial measures group.
A significant difference was found between the importance and use of both:
.
Employee training measures (i.e. in the innovation and learning group), which
include measures such as “quality related training provided to employees”,
“percent of employees who have quality as a major responsibility”, “surveys of
employee satisfaction/attitudes” and “improvement of employee skill/knowledge
levels”.
.
Customer requirement measures (i.e. the customer group), which include
measures such as “ability to adapt or tailor products to customer needs”;
“response time to customer requests for ‘specials’”; and “accuracy of
interpretation of customer requirements”.

Again, there was insufficient evidence to suggest differences in the level of use of
performance measures between industry and service enterprises, and between SMEs.
However, in this sample, medium enterprises make greater use of internal business
process and financial measures while for the small ones it is the use of innovation and
customer measures.
Companies certified to a quality standard and those that were not, did not show any
significant differences between their mean levels of use of performance measures.
Levene’s test for the homogeneity of variances was violated in customer performance
measures. Figure 8 shows this difference in variance, suggesting that SMEs working to
a quality standard are more likely to adopt customer performance measures. A similar
conclusion can be drawn from other measures but this was the only case that was
statistically significant.

Conclusions
The study investigated the current level of knowledge of performance measures and
their degree of implementation in English SMEs. It identified differences between some
groups of companies and presented the biggest obstacles to the introduction of new
measures.
Results indicate that the SMEs surveyed, recognise the importance of the
performance measurement system but their level of use was significantly lower. This
implies that there is a gap between theory and practice, which could be considered an
improvement opportunity for English SMEs.
Performance measures can be used to influence behaviour and, thus, affect the
implementation of strategy (Neely et al., 1994). The OPPMS as part of a
continuous improvement process, linking strategy to results is not balanced,
meaning that this cycle is not fully understood by SMEs’ managers. Although it is
not necessary to use all the measures suggested in the questionnaire, an alignment
 6 Performance
measures in
5
English SMEs
4

3 131
2

1
C_PERF_U

0 40
Figure 8.
Use of customer
–1 performance measures for
N= 17 29 SMEs working/not
No Yes working according to a
quality standard
STANDARD

between strategy and performance measures makes them more effective (McAdam
and Bailie, 2002).
Training of employees and difficulty defining new performance measures were
highlighted as the most important obstacles to the adoption of new performance
measures. This may reflect a lack of skills by employees and a difficulty in
understanding the process. Only a minority of the respondent SMEs were applying
statistical process control and cultural change programmes.
The data collected from this survey suggests that there are no significant
differences in the use of performance measures between industry and service
enterprises, and between SMEs. However, this requires further study, since one
limitation of this study was the low response rate, which precludes a generalisation of
these findings.
Overall, financial measures were the most widely used, while innovation and
learning measures were rated less important and were less used. The most important
performance measures were not consistent with criteria to win new orders.
Based on the data collected, a gap was detected between the importance and use of
some measures suggesting that SMEs should use more productivity, employee
training and customer requirement measures. In particular, the level of use of
innovation and learning measures should increase if SMEs can resolve the major
obstacles, identified in this work, to the adoption of new measures: training of
employees and difficulty defining new measures.
This research is part of a PhD programme to develop a simple and easy-to-use
framework to allow SMEs to create their own performance measurement system,
aligned with strategy, to allow the achievement of pre-determined goals.
BIJ References
13,1/2 Ahmed, A.M. (2002), “Virtual integrated performance measurement”, International Journal of
Quality & Reliability Management, Vol. 19 No. 4, pp. 414-41.
Andersen, H., Cobbold, I. and Lawrie, G. (2001), “Balanced scorecard implementation in SMEs:
reflection on literature and practice”, Proceedings of the Fourth SMESME International
Conference, Aalborg University, Aalborg, Denmark, pp. 103-12.
132 André, M. and Saraiva, P. (2000), “Approaches of Portuguese companies for relating customer
satisfaction with business results”, Total Quality Management, Vol. 11 No. 7, pp. 929-39.
Armstrong, J.S. and Overton, T.S. (1977), “Estimating nonresponse bias in mail surveys”, Journal
of Marketing Research, Vol. 14, pp. 396-402.
Basu, R. (2001), “New criteria of performance management”, Measuring Business Excellence,
Vol. 5 No. 4, pp. 7-12.
Bogan, C.E. and English, M.J. (1994), Benchmarking for Best Practices – Winning through
Innovative Adaptation, McGraw-Hill, New York, NY.
Bourne, M., Mills, J., Wilcox, M., Neely, A. and Platts, K. (2000), “Designing, implementing and
updating performance measurement systems”, International Journal of Operations &
Production Management, Vol. 20 No. 7, pp. 754-71.
Brah, S.A., Tee, S.S.L. and Rao, B. (2002), “Relationship between TQM and performance of
Singapore companies”, International Journal of Quality & Reliability Management, Vol. 19
No. 4, pp. 356-79.
Campanela, J. (Ed.) (1999), Principles of Quality Costs – Principles, Implementation and Use, ASQ
Quality Press, Milwaukee, WI.
Churchil, G.A. (1979), “A paradigm for developing better measures of marketing constructs”,
Journal of Marketing Research, Vol. 16, pp. 64-73.
Cobbold, I.M. and Lawrie, G.J.G. (2002), “The development of the balanced scorecard as a
strategic management tool”, Proceedings of the PMA 2002, Boston, MA, USA, pp. 125-32.
Commission of the European Communities (2003), “Creating an entrepreneurial Europe – the
activities of the European Union for small and medium-sized enterprises (SMEs)”,
Commission of the European Communities, Brussels, available at: http://europa.eu.int/
comm/enterprise/entrepreneurship/promoting_ entrepreneurship/
EFQM (2001), “Moving from the SME model to the EFQM excellence model – SMEs version”,
EFQM, available at: www.efqm.org/publications/downloads/MovingModelsPDF.pdf
EFQM (2003), “Introducing excellence”, available at: www.efqm.org/Downloads/pdf/
0723-InEx-en.pdf
Frigo, M.L. (2002), “Nonfinancial performance measures and strategy execution”, Strategic
Management, August, pp. 6-9.
Hair, J.F., Anderson, R.E., Tatham, R.L. and Black, W.C. (1998), Multivariate Data Analysis,
Prentice-Hall, Englewood Cliffs, NJ.
Hansson, J. and Klefsjo, B. (2003), “A core value model for implementing total quality
management in small organisations”, The TQM Magazine, Vol. 15 No. 2, pp. 71-81.
Hooper, J. (2001), “The process approach to QMS in ISO 9001 and ISO 9004”, Quality Progress,
Vol. 34 No. 12, pp. 70-3.
Hudson, M., Smart, A. and Bourne, M. (2001), “Theory and practice in SME performance
measurement systems”, International Journal of Operations & Production Management,
Vol. 21 No. 8, pp. 1096-115.
ISO (2002), “ISO 9000 – quality management systems”, International Organisation for Performance
Standardization, ISO, Geneva, available at: www.iso.ch/iso/en/iso9000-14000/iso9000/
qmp.html measures in
Ittner, C.D. and Larcker, D.F. (1997), “The performance effects of process management English SMEs
techniques”, Management Science, Vol. 43 No. 4, pp. 522-34.
Juran, J.M. and Godfrey, A.B. (1999), Juran’s Quality Handbook, McGraw-Hill, New York, NY.
Kanji, G. and Sá, P. (2002), “Kanji’s business scorecard”, Total Quality Management, Vol. 13 No. 1, 133
pp. 13-27.
Kaplan, R. and Norton, D. (1992), “The balanced scorecard – measures that drive performance”,
Harvard Business Review, Vol. 70, pp. 71-9.
Kaplan, R. and Norton, D. (1993), “Putting the balanced scorecard to work”, Harvard Business
Review, Vol. 71, pp. 134-47.
Kaplan, R. and Norton, D. (1996), “Using the balanced scorecard as a strategic management
system”, Harvard Business Review, Vol. 74, pp. 75-85.
Karapetrovic, S. and Willborn, W. (2002), “Self-audit of process performance”, International
Journal of Quality & Reliability Management, Vol. 19 No. 1, pp. 24-45.
McAdam, R. and Bailie, B. (2002), “Business performance measures and alignment impact on
strategy”, International Journal of Operations & Production Management, Vol. 22 No. 9,
pp. 972-96.
Neely, A., Adams, C. and Kennerley, M. (2002), The Performance Prism: The Scorecard for
Measuring and Managing Business Success, Financial Times/Prentice-Hall, Harlow.
Neely, A., Gregory, M. and Platts, K. (1995), “Performance measurement system design”,
International Journal of Operations & Production Management, Vol. 15 No. 4, pp. 80-116.
Neely, A., Mills, J., Gregory, M. and Richards, H. (1994), “Realising strategy through
measurement”, International Journal of Operations & Production Management, Vol. 14
No. 3, pp. 140-52.
Neely, A., Richards, H., Mills, H., Platts, K. and Bourne, M. (1997), “Designing performance
measures: a structured approach”, International Journal of Operations & Performance
Management, Vol. 17 No. 11, pp. 1131-52.
Owlia, M.S. and Aspinwall, E.M. (1998), “A framework for measuring quality in engineering
education”, Total Quality Management, Vol. 9 No. 6, pp. 501-18.
Samson, D. and Terziovski, M. (1999), “The relationship between total quality management
practices and operational performance”, Journal of Operations Management, Vol. 17 No. 3,
pp. 393-409.
Saraph, J., Benson, P. and Schroeder, R. (1989), “An instrument for measuring the critical factors
of quality management”, Decision Sciences, Vol. 20 No. 4, pp. 810-29.
Shaffer, R.H. and Thomson, H.A. (1992), “Successful change programs begin with results”,
Harvard Business Review, Vol. 70 No. 1, pp. 80-9.
Sterman, J.D., Repenning, N.P. and Kofman, F. (1997), “Unanticipated side effects of successful
quality programs: exploring a paradox of organizational improvement”, Management
Science, Vol. 43 No. 4, pp. 503-34.
Wilbur, J.H. (2002), “Is time running out for quality”, Quality Progress, Vol. 35 No. 7, pp. 75-9.
Wood, C. and Childe, S. (2003), “Strategic performance measures for business process re-design”,
Proceedings of Business Excellence I – Performance Measures, Benchmarking and Best
Practices in New Economy, University of Minho, Portugal, pp. 102-7.
BIJ Yusof, S.M. and Aspinwall, E.M. (2000a), “TQM implementation issues: review and case
study”, International Journal of Operations & Production Management, Vol. 20 No. 6,
13,1/2 pp. 634-55.
Yusof, S.M. and Aspinwall, E.M. (2000b), “Critical success factors in small and medium
enterprises: survey results”, Total Quality Management, Vol. 11 Nos 4/5 & 6, pp. S448-62.

134 Corresponding author

Sérgio D. Sousa can be contacted at: sds@dps.uminho.pt

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

Performance
Performance measurement of measurement
AMT: a cross-regional study of AMT
Carlo A. Mora Monge
Department of Accounting and BCS, College of Business Administration 135
and Economics, New Mexico State University, Las Cruces,
New Mexico, USA
S. Subba Rao
Department of Management, College of Business Administration,
University of Toledo, Toledo, Ohio, USA
Marvin E. Gonzalez
Department of Management and Marketing,
School of Business and Economics, College of Charleston,
Charleston, South Carolina, USA, and
Amrik S. Sohal
Department of Management, Faculty of Business and Economics,
Monash University, Caulfield East, Australia

Abstract
Purpose – To examine the relationship of performance in advanced manufacturing technologies
(AMT) to the levels of AMT investments and planning and implementation activities in two regions of
America: Anglo-Saxon (USA and Canada) and Hispanic (Mexico and Costa Rica).
Design/methodology/approach – Survey methodology was employed to collect data. The
instrument was translated into Spanish for administration in the Hispanic region. Exploratory factor
analysis was used to establish discriminant validity of the constructs under investigation. Through
multiple regression analysis, predictors for two types of performance (organizational and operational)
were examined.
Findings – Both types of performance are reasonably predicted by the AMT investment and
planning and implementation factors. Performance predictors are different between the two regions.
Research limitations/implications – There are limitations common to survey research
(e.g. subjective perceptions and respondent bias). Also, results depart from the literature in terms of
the predictors for operational and organizational performance. This can be due to other complex
relationships among the variables not identifiable by regression analysis. Future work should address
this by using more sophisticated statistical tools such as structural equation modeling.
Originality/value – The study can help managers understand the factors leading to a successful
AMT implementation. This is one of the first studies on AMT in developing countries; and as such, it
should encourage more research in these countries.
Keywords Advanced manufacturing technologies, Performance measures, North America,
Central America
Paper type Research paper Benchmarking: An International
Journal
Vol. 13 No. 1/2, 2006
pp. 135-146
q Emerald Group Publishing Limited
The authors thank the referees for their insightful comments and suggestions which have 1463-5771
improved both the content and presentation of the paper. DOI 10.1108/14635770610644637
BIJ Introduction
13,1/2 Current global economic conditions are bringing great challenges to firms, which can
affect corporate strategic directions and alter business and manufacturing strategies
(Gordon and Wiseman, 1995; Ferdows, 1997; Sambasivarao and Deshmukh, 1995). In
an effort to survive under such conditions, companies are giving a more strategic role
to manufacturing, from simply supporting marketing strategies to playing a major role
136 in strengthening a company’s market position (Wheelwright and Hayes, 1985). This
change is due in part to the advancement in the automation of manufacturing
technologies. Those technologies are known as advanced manufacturing technologies
(AMT), and Youssef (1992) defined them as a group of integrated hardware and
software-based technologies, which if properly implemented, monitored and evaluated,
can provide greater improvements in efficiency and effectiveness to a firm. In fact,
many companies invest on AMT mainly because they believe it will provide them
simultaneous benefits of faster speed, greater product variety and increased
productivity (Nemetz and Fry, 1988). However, implementation of AMT has been
found to be one of the most lengthy, expensive and complex tasks a firm can undertake
(Sambasivarao and Deshmukh, 1995).
The increasing sophistication of AMTs and the growing importance of having
world-class suppliers are causing more multinational firms to place less emphasis on
low wages when selecting manufacturing sites. Leading manufacturers recognize that
aspects such as labor skills, infrastructure, among others, allows them to be more
competitive on a global basis (Ferdows, 1997).
Understanding the role of AMT and how performance is achieved in different
regions of the world is a paramount. Responding to this need, the current study tries to
enhance the research on AMT by comparing two of the regions in America: an
Anglo-Saxon region, composed by the United States and Canada, and a Hispanic
region, composed by Mexico and Costa Rica. We investigate empirically whether AMT
investment, and AMT planning and implementation activities have an impact on
performance.

Literature review and research objective

Manufacturing literature have emphasized for decades the need to consider the
importance of manufacturing in the strategic decision-making process (Wheelwright,
1984; Wheelwright and Hayes, 1985; Swamidass and Kotha, 1998; Swamidass and
Newell, 1987). Since the early 1980s, firms have applied these concepts and have
realized that competitive capabilities are built upon both structural (e.g. technologies,
processes) and infrastructural aspects (e.g. management, human, training, etc.) (Small
and Yasin, 1997a). In this review, we will discuss relevant structural and
infrastructural aspects of AMT. We begin our discussion by describing different
AMT typologies found in the literature. We then describe and discuss strategic
planning and implementation activities, important to a successful attainment of AMT
benefits. Finally, we describe AMT performance measures.

AMT typologies
Research, in an attempt to understand the gamut of technologies that fall into AMT,
has developed different taxonomies for classification purposes using factor analytic
methods. For example, Chen and Small (1996) grouped AMT in three dimensions:
direct (real time control process, AMHS, barcoding, environmental control systems, Performance
FMS, automated inspection, robotics, GT), indirect (CAD, CNC, CAE, CAM), and measurement
administrative (ABC, DSS, electronic mail, KBS, office automation, MRP, MRP II, EDI).
This classification has been empirically validated by Beaumont et al. (2002), and we of AMT
use it as our basis in our study. Table I presents additional classifications found in the
literature.
137
AMT planning and implementation activities
Theoretical and empirical evidence suggest that planning and implementation
activities are critical to achieve all the benefits of AMT. In fact, inadequate planning
and implementation are two of the most important reasons why AMT projects fail
(Saraph and Sebastian, 1992; Boyer et al., 1997; Co et al., 1998; Millen and Sohal, 1998).
Research has identified several factors required for proper planning and
implementation of AMT. Co et al. (1998) contend that although AMT requires a
high level of technical expertise, management aspects play a critical role. In fact, a lack
of an appropriate organizational infrastructure proves to be the greatest impediment to
the effective implementation of new manufacturing technologies (Chen and Small,
1996). Research suggests that most of the benefits of AMT result from the new form of
organization which AMT requires, rather than the technology itself (Chen and Small,
1996). Nemetz and Fry (1988) argue that to fully exploit the benefits of AMT, a flexible
organization capable of quickly react to changes is needed. This implies that
organizational structures should become more organic characterized by a more
responsive decision-making processes, greater participation by workers, smaller
differences between workers and managers, higher levels of communication, and
higher levels of horizontal interaction (Saraph and Sebastian, 1992).
Another factor discussed in the literature is the linkage between functional areas
including marketing, R&D, finance, and manufacturing (Doll and Vonderembse, 1987;
Wheelwright and Hayes, 1985; Sohal, 1996; Maffei and Meredith, 1994). The
organization as a whole should have a clear understanding of the objectives and
implications of an AMT project, and the strategies should be aligned towards

Author(s) Dimensions

Adler (1988) and Boyer et al. (1996)
Rosenthal (1984)
Swamidass and Kotha (1998)
Small and Yasin (1997b)
Small and Chen (1997)
Chen and Small (1996) and
Beaumont et al. (2002)
Design, manufacturing, and administrative
Computer-aided design, computer-aided
manufacturing, and factory management and control
group
Information exchange and planning technology,
production design technology, high-volume
automation technology, and low-volume flexible
automation technology
Stand-alone, intermediate, and integrated systems
Design and engineering, fabricating/machining and
assembly technologies, automated inspection and
testing equipment, and information technologies Table I.
Direct, indirect, and administrative AMT investment
dimensions
BIJ obtaining those objectives. This requires the participation of not only all members in
13,1/2 the organization, but also external players (Co et al., 1998).
It is tempting to believe that in an automated environment, human intervention is
not that important, but as research has found, just the opposite is true (Maffei and
Meredith, 1994). AMT requires multi-skilled workers (Maffei and Meredith, 1994;
Saraph and Sebastian, 1992) and teamwork (Maffei and Meredith, 1994). The
138 importance of teams is stressed by Gordon and Wiseman (1995), who found that on
average exceptional plants have more than half of the production workforce
participating in teams. Teams ought to have a multidisciplinary structure (Doll and
Vonderembse, 1987; Gerwin and Kolodny, 1992). The significance of multifunctional
focus is consistent with findings suggesting that cross-functional teams are a
necessary component for successful AMT implementation (Hottenstein et al., 1997; Co
et al., 1998; Sohal, 1996). Teams should be led by a project champion, who takes a key
role in the team (Chen and Small, 1996).
Channels of communication are also needed to attain the expected benefits (Maffei
and Meredith, 1994; Saraph and Sebastian, 1992).
Personnel training is another factor mentioned in the literature (Adler, 1988; Co et al.,
1998). Hottenstein et al. (1997) found that the existence of an employee educational
program prior to AMT implementation as a significant factor in distinguishing
between a successful and unsuccessful project.
In summary, although AMT can be a strategic weapon, it continues to be a
risky and complex task to undertake (Frohlich, 1998), because it not only high
financial investments, but also aspects such as internal and external planning, and
team-based activities (TBA) (Boyer et al., 1997; Co et al., 1998; Millen and Sohal,
1998).

AMT performance
Literature has identified different dimensions for measuring performance and we
summarize them in Table II. Small and Yasin (1997b) measured performance from
internal and external perspectives, while Beaumont et al. (2002) and Ramamurthy
(1995) provide a two-dimensional approach for performance measurement: operational
and organizational performance (ORGP). Operational performance (OPEP) deal with
short term and tangible measures such as cost, flexibility, and quality, while ORGP
measures are less tangible (e.g. enhanced firm’s image, improved market share,
product differentiation through innovation, etc.). In this research, we use both
operational and organizational measures, as suggested by Beaumont et al. (2002).

Author(s) Dimensions

Adler (1988), Sohal (1991) and Cost, quality, flexibility, and

Small and Chen (1995) time-based competition
Small and Yasin (1997b) Internal and external
Udo et al. (1995) Technical success and business success
Table II. Udo et al. (1995) and Sohal et al. (1999) Tangible and intangible
AMT performance Ramamurthy (1995) and Beaumont et al. (2002) Operational and organizational
Research objective Performance
Literature suggests that AMT performance is a function of both technological and measurement
planning issues. This study explores the above aspect by analyzing the effect of AMT
investments and planning and implementation activities on AMT performance. We of AMT
also investigate whether predictors of AMT performance are different between the
Anglo-Saxon and Hispanic regions.
139
Methodology
Measurement instrument
The instrument used in the current study is part of a world wide research on AMT
initiated by Sohal (1991), and includes five sections: company details, AMT
investments, planning, implementation and assessment of AMT investments,
performance, and lessons learned.
The instrument has been used in several countries including Australia (Sohal, 1996),
New Zealand (Schroder and Sohal, 1999), Canada (Beaumont et al., 2002), and the
United States (Millen and Sohal, 1998). This is the first time the instrument is being
used in Mexico and Costa Rica. Because Spanish is the official language in these
countries, the instrument was translated by one of the authors, a native Spanish
speaker, and reviewed by a professional translator with experience in the field.
Practitioners and academicians from both countries checked the Spanish version for
terminology and context and minor modifications were made in the final version of the
questionnaire. This approach is consistent with previous works in cross-national
research (Soutar et al., 1999; Cadogan et al., 1999).

Procedure and sample

The sample for the Anglo-Saxon region consisted of 97 firms. In Canada, the
information was obtained through a mail survey by a major university in that country.
In the United States, the authors collected the sample using a mailing list similar to the
one used by Boyer et al. (1997). The SIC codes considered in the study are from 33 to 37
which are considered to be the industries with the most wide use of AMT (Boyer et al.,
1997).
The sample for the Hispanic region consisted of 125 firms. Out of these, 85 represent
the Mexican firms and the remaining 40 Costa Rican. The respondents to the survey in
this region were dram from companies with similar characteristics to the Anglo-Saxon
counterpart, that is, companies likely to have AMT investments.
Accompanying the questionnaire, a cover letter was included that explained the
purpose of the study, and assured anonymity to increase the objectivity of their
responses.
Potential confounding effects and possible difference between the countries and
regions were addressed by evaluating company size. Other studies have shown that
size is related to AMT adoption (Lefebvre et al., 1996; Swamidass and Kotha, 1998). For
the countries, we conducted a one-way ANOVA test that confirmed non-significance in
terms of company size. Scheffe post-hoc tests indicated non-significance for all pair of
countries (CAN-MEX, p ¼ 0.978; CAN-CRI, p ¼ 0.969; CAN-USA, p ¼ 0.066;
MEX-CRI, p ¼ 0.999; MEX-USA, p ¼ 0.112; CRI-USA, p ¼ 0.250). This gives
confidence that the samples could be combined, and furthermore, that differences
that might exist will not be caused by size.
BIJ Measurement and variables
13,1/2 Using exploratory factor analysis (EFA) with varimax rotation, we established
discriminant validity of the constructs. As a general rule of thumb, when the sample
size is 50 or larger, factor loadings of greater than 0.50 are very significant (Hair et al.,
1995). A scale with good internal consistency should have all items load on one factor.
If multiple factors emerge, the possibility of splitting the items into multiple
140 dimensions has to be examined, and theoretical justifications should be sought. We
utilized the factor analysis with varimax rotation method, which focuses in simplifying
the columns of the factor matrix, thus giving a clearer separation of the factors
compared to other rotation methods (Hair et al., 1995).
AMT investments. The survey measured the AMT investments using 21 items
using a seven-point Likert scale (no investment to heavy investment). These measures
have been validated earlier (Beaumont et al., 2002), where the authors found three
factors: direct (technology used on the factory floor); indirect (technology used to
design products and schedule production); and administrative (technologies used to
give administrative support to the factory and integrate its operations with the rest of
the organization).
The results are shown in Table III. Four clear factors emerged, all with factor
loadings above 0.5. Two of the factors are related to shop floor technologies and two to
administrative technologies. In terms of the shop floor related factors, we labeled them
as DMT (direct manufacturing technologies, technologies directly involved in the

Factor
Item DMT AIET IMT APT
a
Direct manufacturing technologies (DMT) (0.85)
Computer-aided design (CAD) 0.816
Computer-aided engineering (CAE) 0.796
Numerical control machinery (CNC) 0.741
Computer-aided manufacturing (CAM) 0.672
Robotics 0.552
Administrative-information exchange –
strategic technologies (AIET) (0.87) a
Electronic data interchange (EDI) 0.835
Decision support systems (DSS) 0.777
Electronic mail 0.777
Knowledge-based systems (KBS) 0.684
Integrative manufacturing technologies (IMT) (0.76) a
Automated materials handling systems (AMHS) 0.736
Real-time process control systems 0.727
Barcoding/automatic identification 0.704
Automated inspection 0.601
Flexible manufacturing systems (FMS) 0.577
Administrative-planning technologies (APT) (0.89) a
Manufacturing resource planning (MRP II) 0.727
Materials requirements planning (MRP) 0.726
Eigenvalue 6.80 1.60 1.38 1.04
Table III.
Percent of variance explained (total ¼ 67.6 percent) 42.5 10.0 8.6 6.5
Factor analysis – AMT
technologies Notes: aCronbach a reliability coefficients. Only loadings of greater than 0.5 are presented
design and production processes) and IMT (integrative manufacturing technologies, Performance
technologies involved in the integration of different processes). This finding is similar measurement
to empirical evidence found by Swamidass and Kotha (1998), who found more than one
dimension for shop floor technologies. With regards to the administrative technologies of AMT
factors, we labeled them as AIET (administrative-information exchange- strategic
technologies, technologies used for the exchange of information, such as electronic mail
and EDI, and technologies that play an important role in the decision-making process, 141
such as DSS and KBS), and APT (administrative-planning technologies, technologies
utilized in the planning process). The factors account for almost 68 percent of the
variance, which is considered to acceptable.
Planning and implementation activities. Planning and implementation of AMT was
operationalized through 22 items in the survey, including a wide range of areas such as
strategic, organizational, and technology. These items were measured using a
seven-point Likert scale where 1 represented no efforts and 7 maximum efforts put in a
particular activity. Some of the items included in the survey have been empirically
validated in previous research (Small and Yasin, 1997a). Table IV shows three
underlying factors that account for 65 percent of the variance and all items have
loadings 0.50 or greater. The first factor, TBA includes activities that deal with group
activities, like need for multi-disciplinary implementation teams, communication, and
the importance of having multi-skilled workers. The second factor, IPA (internal
planning activities) relates to activities internal to the organization such as linking
business and manufacturing strategies, strategic investment, and the need for having

Factor
Item TBA IPA EPA

Team-based activities (TBA) (0.83) a

Emphasizing teamwork and group activities 0.832
Having multi-skilled production workers 0.793
Communicating the likely impact of new AMT to all 0.777
plant workers
Establishing multi-disciplinary implementation 0.507
teams
Internal planning activities (IPA) (0.83) a
Financial investment evaluation prior to installation 0.872
Choosing knowledgeable project leaders 0.735
Strategic investment evaluation prior to installation 0.682
Pre-installation training of all project participants 0.554
Linking manufacturing to business strategy 0.505
External planning activities (EPA) (0.77) a
Obtaining the services of knowledgeable AMT 0.842
consultants
Hiring or retaining AMT experts or plant staff 0.771
Monitoring AMT being used in the core industry 0.605
Establishing multi-disciplinary planning teams 0.565
Eigenvalue 6.01 1.32 1.06 Table IV.
Percent of variance explained (total ¼ 64.5 percent) 46.2 10.2 8.1 Factor analysis –
planning and
Notes: aCronbach a reliability coefficients. Only loadings of greater than 0.5 are presented implementation activities
BIJ knowledgeable project leaders. The third factor, EPA (external planning activities),
13,1/2 relates to planning activities that have implications external to the firm such as
monitoring the industry, use of consultants, and hiring or retaining AMT experts
loaded in this factor.
Performance. The survey included nine items, using a seven-point Likert scale,
measuring the extent to which AMT had enabled the firm to improve its performance.
142 These measures have been previously validated by Beaumont et al. (2002), who showed
two underlying performance factors: operational (more tangible type of measures such
as reduction in costs, enhanced productivity, etc.) and organizational (considered to be
more intangible and include enhancement of firm’s image, increase in market share,
etc.). In this research, we validated these factors with a different sample. The results
(Table V) show two factors that account for 63 percent of the total variance. We labeled
them as OPEP and ORGP.

Results and discussion

To explore the relationship between performance and the level of AMT investments
and planning and implementation activities in the two regions, we use multiple linear
regression analysis. We use two dependent variables OPEP and ORGP, and regress
them on the independent variables of AMT investments factors (DMT, IMT, AIET,
and APT) and AMT planning and implementation factors (TBA, IPA, and EPA).
Additionally, we use an indicator variable to represent the region (0 – Anglo-Saxon
and 1 – Hispanic). The multiple regression model is as follows:

Factor
Item OPEP ORGP

Operational performance (OPEP) (0.85) a

The degree to which enhancement of market share 0.826
has been facilitated
Competitors find it difficult to imitate our innovation 0.777
The extent to which the image of the firm as a 0.740
sophisticated and progressive manufacturer has
been fostered/enhanced by the use of AMT
innovation
The extent to which customers/market have become 0.719
more dependent on/committed to the company
The extent to which organizational flexibility in 0.647
responding to market changes has been enhanced
Organizational performance (ORGP) (0.82) a
The degree to which consistency of the product and 0.913
management of quality has improved
The extent to which productivity has been enhanced 0.868
Switching costs have decreased 0.703
Management has better control over manufacturing 0.549
process
Eigenvalue 4.27 1.41
Table V. Percent of variance explained (total ¼ 63.1 percent) 47.5 15.6
Factor analysis –
performance measures Notes: aCronbach a reliability coefficients. Only loadings of greater than 0.5 are presented
 Performance ¼ b0 þ b1 DMT þ b2 IMT þ b3 AIET þ b4 APT þ b5 TBA þ b6 IPA Performance
measurement
þ b7 EPA þ b8 D
of AMT
Table VI shows the results of the multiple regression analysis. Firstly, the indicator
variable is found to be significant, implying that the relationship between performance
and the independent variables is different for the two regions. Secondly, we can see 143
that both operational and ORGP of AMT are reasonably predicted by the level of AMT
investment and AMT planning and implementation factors (R 2 values varying from
26 to 39 percent). This confirms our hypothesis that performance is positively related
to AMT Investments and planning and implementation activities. We have also shown
that performance varies by the region.
There are interesting findings noteworthy mentioning. The analysis indicates
positive relationships between DMT and IMT on ORGP. With OPEP considered as the
dependent variable, IPA and TBA emerged as significant predictors. These findings
are somewhat unexpected with respect to previous research (Swamidass and Kotha,
1998; Udo and Ehie, 1996; Zammuto and O’Connor, 1992).

Conclusions
The present international study explored the relationship between AMT investment,
implementation and planning, and performance in two regions of America. It is hoped
that the results of this study will add to the growing body of knowledge on AMT in
different ways. First, this is one of the first studies on AMT in developing countries;
and as such, it should encourage more research in these countries to analyze in greater
detail implementation, and performance measurement issues in less developed
countries. Second, we validated some research constructs that might be used in future
international research. Third, we found empirical evidence that predictors of
performance vary according to the region.
Although our study has brought out some interesting findings, there are some
limitations common to survey studies. The results are based on the subjective
perception of the respondents; therefore, generalization of results should be done
carefully (Udo and Ehie, 1996). There might also be some bias by having the same
respondent reporting on both predictor and outcome variables, hence; our findings
should be interpreted within these limitations. As stated by Ramamurthy (1995), using
a questionnaire-based field survey may not have adequately capture the complex
dynamics that usually accompany a major strategic decision such as an adoption and
implementation of AMT.
Finally, although literature suggests positive relationships between AMT use and
OPEP (Swamidass and Kotha, 1998), and planning and implementation activities and

Dependent
variable Constant TBA IPA EPA APT AIET DMT IMT Dummy R2

OPEP 3.562 * * 0.188 * * 0.150 * * 20.050 0.027 20.089 20.007 0.104 0.593 * * 0.256
ORGP 2.039 * * 0.076 0.159 0.057 20.026 20.094 0.278 * * 0.147 * 0.882 * * 0.391 Table VI.
Multiple regression
Notes: *Significant at 0.05 level; * *significant at 0.01 level results
BIJ ORGP (Udo and Ehie, 1996; Zammuto and O’Connor, 1992), our results indicated just
13,1/2 the opposite. It is possible that there are confounding effects, interactions effects, and
other complex relationships among the variables leading to the current results. This
needs to be addressed in future work through the use of more sophisticated statistical
tools such as structural equation modeling.

144 References
Adler, P.S. (1988), “Managing flexible automation”, California Management Review, Vol. 30 No. 3,
pp. 34-56.
Beaumont, N., Schroder, R. and Sohal, A. (2002), “Do foreign-owned firms manage advanced
manufacturing technology better?”, International Journal of Operations & Production
Management, Vol. 22 No. 7, pp. 759-71.
Boyer, K.K., Leong, G.K., Ward, P.T. and Krajewski, L.J. (1997), “Unlocking the potential of
advanced manufacturing technologies”, Journal of Operations Management, Vol. 15 No. 4,
pp. 331-47.
Boyer, K.K., Ward, P.T. and Leong, G.K. (1996), “Approaches to the factory of the future – an
empirical taxonomy”, Journal of Operations Management, Vol. 14 No. 4, pp. 297-313.
Cadogan, J.W., Diamantopoulos, A. and de Mortanges, C.P. (1999), “A measure of export market
orientation: scale development and cross-cultural validation”, Journal of International
Business Studies, Vol. 30 No. 4, pp. 689-707.
Chen, I.J. and Small, M.H. (1996), “Planning for advanced manufacturing technology: a research
framework”, International Journal of Operations & Production Management, Vol. 16 No. 5,
pp. 4-24.
Co, H.C., Patuwo, B.E. and Hu, M.Y. (1998), “The human factor in advanced manufacturing
technology adoption: an empirical analysis”, International Journal of Operations &
Production Management, Vol. 18 No. 1, pp. 87-106.
Doll, W. and Vonderembse, M. (1987), “Forging a partnership to achieve competitive advantage:
the CIM challenge”, MIS Quarterly, Vol. 11 No. 2, pp. 205-20.
Ferdows, K. (1997), “Making the most of foreign factories”, Harvard Business Review, Vol. 75
No. 2, pp. 73-88.
Frohlich, M. (1998), “How do you successfully adopt an advanced manufacturing technology?”,
European Management Journal, Vol. 16 No. 2, pp. 151-9.
Gerwin, D. and Kolodny, H. (1992), Management of Advanced Manufacturing Technology:
Strategy, Organization & Innovation, Wiley, New York, NY.
Gordon, J. and Wiseman, J. (1995), “Thriving on competition”, Ivey Business Quarterly, Vol. 59
No. 3, pp. 79-84.
Hair, J.F., Anderson, R.E., Tatham, R.L. and Balck, W.C. (1995), Multivariate Data Analysis,
Prentice-Hall, Upper Saddle River, NJ.
Hottenstein, M.P., Casey, M.S. and Dunn, S.C. (1997), “Facilitation of advanced manufacturing
technology: implementation and transfer”, Industrial Management, Vol. 39 No. 5, pp. 8-13.
Lefebvre, L.A., Lefebvre, E. and Harvey, J. (1996), “Intangible assets as determinants of advanced
manufacturing technology adoption in SMEs: toward an evolutionary model”, IEEE
Transactions on Engineering Management, Vol. 43 No. 3, pp. 307-20.
Maffei, M.J. and Meredith, J. (1994), “The organizational side of flexible manufacturing
technology”, International Journal of Operations & Production Management, Vol. 14 No. 8,
pp. 17-34.
Millen, R. and Sohal, A.S. (1998), “Planning processes for advanced manufacturing technology by Performance
large American manufacturers”, Technovation, Vol. 18 No. 12, pp. 741-50.
Nemetz, P.L. and Fry, L.W. (1988), “Flexible manufacturing organizations: implication for
measurement
strategy formulation and organization design”, Academy of Management Review, Vol. 13 of AMT
No. 4, pp. 627-38.
Ramamurthy, K. (1995), “The influence of planning on implementation success of advanced
manufacturing technologies”, IEEE Transactions on Engineering Management, Vol. 42 145
No. 1, pp. 62-73.
Rosenthal, S.R. (1984), “Progress toward the ‘factory of the future’”, Journal of Operations
Management, Vol. 4 No. 3, pp. 203-29.
Sambasivarao, K.V. and Deshmukh, S.G. (1995), “Selection and implementation of advanced
manufacturing technologies: classification and literature review of issues”, International
Journal of Operations & Production Management, Vol. 15 No. 10, pp. 43-62.
Saraph, J.V. and Sebastian, R.J. (1992), “Human resource strategies for effective introduction of
advanced manufacturing technologies (AMT)”, Production & Inventory Management
Journal, Vol. 33 No. 1, pp. 64-70.
Schroder, R. and Sohal, A.S. (1999), “Organisational characteristics associated with AMT
adoption: towards a contingency framework”, International Journal of Operations &
Production Management, Vol. 19 No. 12, pp. 1270-91.
Small, M.H. and Chen, I.J. (1995), “Investment justification of advanced manufacturing
technology: an empirical analysis”, Journal of Engineering & Technology Management,
Vol. 12 Nos 1/2, pp. 27-55.
Small, M.H. and Chen, I.J. (1997), “Economic and strategic justification of AMT – inferences from
industrial practices”, International Journal of Production Economics, Vol. 49 No. 1,
pp. 65-75.
Small, M.H. and Yasin, M.M. (1997a), “Advanced manufacturing technology: implementation
policy and performance”, Journal of Operations Management, Vol. 15 No. 4, pp. 349-70.
Small, M.H. and Yasin, M.M. (1997b), “Developing a framework for the effective planning and
implementation of advanced manufacturing technology”, International Journal of
Operations & Production Management, Vol. 17 No. 5, pp. 468-89.
Sohal, A.S. (1991), “Investing in AMT”, Australian Accountant, Vol. 61 No. 2, pp. 40-4.
Sohal, A.S. (1996), “Assessing AMT implementations: an empirical field study”, Technovation,
Vol. 16 No. 8, pp. 377-84.
Sohal, A.S., Gordon, J., Fuller, G. and Simon, A. (1999), “Manufacturing practices and competitive
capability: an Australian study”, Technovation, Vol. 19 No. 5, pp. 295-304.
Soutar, G.N., Grainger, R. and Hedges, P. (1999), “Australian and Japanese value stereotypes: a
two country study”, Journal of International Business Studies, Vol. 30 No. 1, p. 203.
Swamidass, P.M. and Kotha, S. (1998), “Explaining manufacturing technology use, firm size and
performance using a multidimensional view of technology”, Journal of Operations
Management, Vol. 17 No. 1, pp. 23-37.
Swamidass, P.M. and Newell, W.T. (1987), “Manufacturing strategy, environment uncertainty
and performance: a path analytical model”, Management Science, Vol. 33 No. 4, pp. 509-24.
Udo, G.J. and Ehie, I.C. (1996), “Advanced manufacturing technologies: determinants of
implementation success”, International Journal of Operations & Production Management,
Vol. 16 No. 12, pp. 6-26.
Udo, G.J., Ehie, I.C. and Olorunniwo, F. (1995), “Fulfilling the promises of advanced
manufacturing systems”, Industrial Management, Vol. 37 No. 5, pp. 23-8.
BIJ Wheelwright, S.C. (1984), “Manufacturing strategy: defining the missing link”, Strategic
Management Journal, Vol. 5 No. 1, pp. 77-91.
13,1/2 Wheelwright, S.C. and Hayes, R.H. (1985), “Competing through manufacturing”, Harvard
Business Review, Vol. 63 No. 1, pp. 99-109.
Youssef, M.A. (1992), “Getting to know advanced manufacturing technologies”, IIE Solutions,
Vol. 24 No. 2, pp. 40-2.
146 Zammuto, R.F. and O’Connor, E.J. (1992), “Gaining advanced manufacturing technologies’
benefits: the roles of organization design and culture”, Academy of Management, Vol. 17
No. 4, pp. 701-28.

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

Benchmarking
Benchmarking performance performance
indicators for banks indicators
Chien-Ta Ho
Department of International Trade, Lan Yang Institute of Technology, 147
Taiwan, Republic of China, and
Yun-Shan Wu
Department of Accounting, Lan Yang Institute of Technology,
Taiwan, Republic of China

Abstract
Purpose – The aim of this paper is to construct a performance evaluation of banks with the stock
market taken into consideration.
Design/methodology/approach – Grey relation analysis (GRA), a concept borrowed from the
study of industry and increasingly applied to commerce, is used to evaluate the relative performance of
three major banks in Australia. The purpose of using GRA is to reduce the number of financial
indicators by selecting representative indicators from financial statement analysis.
Findings – Benchmarking performance indicators are essentially finding the representative
indicator from the existing ratios most commonly used in financial analysis to assess business
operational performance. The paper compares the GRA results from the financial statement analyses
and shows the same result can be obtained.
Research limitations/implications – This paper conducted a review of literature and five-power
analysis to aggregate financial ratios appropriate for the analysis. This method may result in
incompleteness in the aggregation of ratios, and requires adjustment when other issues for analysis
are involved. Future research could set up a specific model for the preliminary selection of financial
ratios with a new to make studies of this kind more complete.
Originality/value – This paper introduced a new approach for performance evaluation – GRA. The
major contribution of this paper is the use of GRA methodology to retrieve ratios most commonly used
in financial analysis to tackle the problems of sample size and distribution uncertainty. This could
avoid the waste of resources due to the uncertainty of relations among the ratios when using them for
analysis.
Keywords Benchmarking, Performance measures, Banks, Financial reporting, Australia
Paper type Research paper

Introduction
If the management of enterprises could conduct their own measurement or diagnosis of
the enterprises at regular intervals, management most likely would gain a better
understanding of how effectively they have used their resources. The result of this
performance evaluation could serve as reference for them in their future allocation of
resources for the enterprise. Benchmarking is undoubtedly a very important tool for
enterprises and consultants for establishing their goals, developing methods for
achieving the goals, and measuring system performance (Tapas, 1998). Therefore, Benchmarking: An International
effective benchmarking performance evaluation for enterprises could definitely Journal
Vol. 13 No. 1/2, 2006
contribute to effective management. pp. 147-159
Performance evaluation of financial institutions, particularly commercial banks, has q Emerald Group Publishing Limited
1463-5771
received increased attention over the past several years (Seiford and Zhu, 1999). Also, DOI 10.1108/14635770610644646
BIJ most previous studies concerning bank performance evaluation focus merely on
13,1/2 operational performance. Stock performance, however, which might directly influence
the decision-making of investors, is usually ignored. As a result, this paper tries to
construct a performance evaluation for Australia’s three major banks with the stock
performance taken into consideration.
The evaluation of operational performance has always been a major concern for
148 industries, governments and academe. Operational performance not only serves as a
basis for organizational improvement and criteria for detecting problems in
enterprises, but also as a policy determinant for governments in mapping out
relevant measures. However, choosing a viable method for effective evaluation of
performance is not an easy task. There is a substantial body of literature discussing
different research methods applied to performance evaluation. These methods include:
. multivariate statistical analysis (Huang, 1986; Fielding et al., 1985);
. data envelopment analysis (Seiford and Zhu, 1999; Lin, 1998; Grosskopf and
Valdmanis, 1987);
.
analytic hierarchy process (Lin, 2000; Shih, 2000);
.
fuzzy set theory (Shih, 2000; Tsai, 2000);
.
grey relation analysis (GRA) (Tsai, 2000);
.
balanced scorecard (Maisel, 1992); and
.
financial statement analysis (FSA) (Collins, 1980; Pantalone and Platt, 1987;
Espahbodi, 1991).

Some of the methods may have already been known to the public. Other methods were
simply borrowed from the domain of industrial study and applied to commerce. Some
are still in the embryonic stage.
Each of the above seven methods can be independently applied to evaluating
performance. However, no one of them is perfect. Researchers can only choose a
method to evaluate performance that has the least amount of drawbacks for that
study’s particular situation. Therefore, a viable method for effective evaluation of
performance is aimed at providing solutions for issues with multiple variables and
targets. In studying Australia’s major banks, financial data are often incomplete or
unclear, and this paper, therefore, is bound by realistic limits, confining itself to a
situation where the amount of data are small and its significance indefinite. Given
these considerations, GRA may be the best method for this study, and will be
used to retrieve ratios most commonly used in financial analysis to tackle the
problems of sample size and distribution uncertainty. Furthermore, TOPSIS
(technique for order preference by similarity to ideal solution) is used to conduct
ranking on the subject of study in order to find the relative performance of the
companies under scrutiny.
The remainder of the paper is organized as follows. Methodology: relation analysis
section introduces GRA. The following section is an empirical study using the example
of Australia’s three major banks. The next section compares GRA results to the
financial statement analysis. The conclusions and suggestions for future investigation
are discussed in conclusion and future research section.
Methodology: grey relation analysis Benchmarking
The purpose of using GRA performance
If all performance indicators in the set are placed into the evaluation process, data
collection would be more difficult and resources would be wasted. Benchmarking indicators
performance indicators are essentially finding the representative indicator from the
existing ratios most commonly used in financial analysis to assess business operational
performance. The purpose of using GRA in the paper is to reduce the number of 149
indicators by selecting representative indicators from among them. In general, the
representative indicators can be selected by clustering, which minimizes the difference
within a certain cluster, and maximizes the differences between those clusters.
In the current study, the cluster was classified in accordance with their respective
attributes under the five-power analysis[1] and stock performance criteria commonly used
in financial analysis: liquidity, asset utilization, leverage, profitability, growth and stock
performance. When the amount of sample data is large enough and it conforms to normal
distribution, then most researchers use the mathematical statistical method (factor
analysis, cluster analysis, discriminate analysis and regression analysis) to conduct the
selection of representative indicators. However, in the current study of three Australia’s
banks, data are often incomplete or unclear, and this paper, therefore, is bound by realistic
limits, confining itself to a situation where the amount of data are small and its significance
indefinite. This paper intends to use the work of Deng (1982) who proposed the selection of
representative indicators based on the GRA. The introduction of GRA is as follows.

GRA
GRA was first proposed by Deng (1982), and was based on the theory of grey relation
space. The fundamental definition of “greyness” is information being incomplete or
unknown, thus an element from an incomplete message is considered a grey element.
Grey relation means the measurement of changing relations between two systems or
between two elements that occur in a system over time (Shih et al., 1994), and GRA is a
research method used to measure the relationships among elements when the trends of
their development have either homogeneity or heterogeneity (Deng, 1989). If two
elements develop in a consistent trend, the two elements have a high level of relation. If
two elements develop in an inconsistent trend, they have a low level of relation. The
definition and modelling of this as follows:
According to Deng (1989), let X ¼ {xj j j ¼ 1; 2; . . . ; n} as factors in grey relations
in sequential order, x1 [ X as reference sequence; xj [ Xð j – 1Þ as comparative
sequence. Then x1(i ) and xj(i ) ði ¼ 1; 2; . . . ; m; j ¼ 2; 3; . . . ; nÞ would be the values of
x1 and xj at point i. If g(x1(i ), xj(i )) are real numbers, then it can be defined as:

1X m
gðx1 ; xj Þ ¼ gðx1 ði Þ; xj ði ÞÞ ð1Þ
m i¼1

The mean of g(x1(i ), xj(i )) need to meet the four axioms of grey relation:
Axiom 1. Norm interval
0 , gðx1 ; xj Þ # 1
gðx1 ; xj Þ ¼ 1 $ x1 ¼ xj ; This is called complete relation
gðx1 ; xj Þ ¼ 0 $ x1 ¼ xj ; This is called complete non-relation
BIJ Axiom 2. Duality symmetric
13,1/2 x; y [ X
gðx; yÞ ¼ gð y; xÞ $ X ¼ ðx; yÞ
Axiom 3. Wholeness
xj ; xi [ X
150 gðxj ; xi Þ – gðxi ; xj Þ
Axiom 4. Approachability
gðxi ði Þ 2 xj ði ÞÞ get larger along with jxi ði Þ 2 xj ði Þj get smaller
If the above-mentioned public theories could all be satisfied, g(x1(i ) and xj(i )) is said to
be the grey relational coefficient of xj to x1 at point i. In the current study, the sequence
of financial indicators in a cluster is based on the magnitude of the grey relational
coefficient. Deng (1982) has proposed a mathematical equation that will satisfy these
four axioms of grey relation, which is as follows:
min min jx1 ði Þ 2 xj ði Þj þ z max max jx1 ði Þ 2 xj ði Þj
j i j i
gðx1 ði Þ; xj ði ÞÞ ¼ ð2Þ
jx1 ði Þ 2 xj ði Þj þ z max max jx1 ði Þ 2 xj ði Þj
j i

g(x1(i ), xj(i )) is designed as the grade of grey relation in xj correspondence to x1 g(x1(i ),

xj(i )) is said to be the grey relational coefficient of xj to x1 at point i, where z [ ½0; 1
is
the distinguished coefficient, the function of which is to reduce its numerical value by:
max max jðx1 ði Þ; xj ði ÞÞj
j i

getting large, so as to effect its loss-authenticity and to heighten the remarkable

difference among relation coefficients.
Empirical study
Data and sample
This empirical study is conducted using the example of Australia’s three[2] major
banks. They are Commonwealth Bank of Australia, National Australia Bank and
Westpac Bank. The data sources of each commercial bank included all the financial
documents in FY 2000. All financial data of these banks are obtained from World
Database of Thomson Financial (as shown in Table I).

Performance indicators set

In the study the authors attempted to adopt FSA (five-power analysis and stock
performance) to select different ratios, the ratios were classified in accordance with their
respective attributes under the six categories: profitability, asset utilization, leverage,
liquidity, growth and stock performance. The authors selected 59 financial ratios as the
aggregated indicators for evaluating the performance of the banks as shown in Table I.
There are 13 ratios for analysing the profitability factor, 16 for leverage, eight for
liquidity, two for asset utilization, 12 for growth and eight for stock performance.

Calculating the coefficient of grey relation and representative indicators

For convenience of calculating the coefficient of grey relation among the financial
ratios, this study used Turbo Pascal 7.0 for the calculation. Vector normalization
 Benchmarking
Ratio Code Indicator CBA NAB WBC
performance
A Profitability A1 Return on investment capital 12.09 6.44 5.23 indicators
A2 ROE per share 39.58 20.21 19.29
A3 Return on equity 40.09 20.44 20.11
A4 Return on assets 2.3 1.83 1.46
A5 Oper profit margin 22.64 21.09 18.79 151
A6 Cash earnings ROE 47.13 2 3.99 14.18
A7 Int income/earn asset 3.24 3.90 3.27
A8 Int income/avg dep 3.64 4.79 4.54
A9 Rtrn on earn asset 2.35 1.69 1.51
A10 Non-int inc/tot revs 22.12 22.44 18.71
A11 Tot int exp/int liab 4.65 5.75 6.15
A12 Tot int inc/ear asset 7.69 9.71 9.07
A13 Effic of earng asset 7.82 9.97 9.19
B Asset utilization B1 Assets/empl 4.95 6.82 4.87
B2 Asset turnover 0.06 0.07 0.08
C Leverage ratio C1 Tot debt/com eqty 206.92 394.69 392.02
C2 Lt debt/com eqty 81.99 233.78 226.51
C3 Tot debt/tot cap 66.95 77.5 78.82
C4 Min int/tot cap 1.17 0.07 0.02
C5 Pfd stock/tot cap 0 4.11 1.59
C6 Tot debt/tot asset 17.58 22.99 22.72
C7 Dividend payout 32.67 46 44.37
C8 Oper cash/fixed chg 8.33 2 0.42 4.48
C9 Fxd asset/com eqty 6.14 13.04 13.36
C10 Lt debt/total cap 44.53 67.11 68.26
C11 Equity/total cap 54.31 28.71 30.13
C12 Com eqty/tot asset 8.5 5.83 5.8
C13 Tot cap/assets 15.64 20.29 19.24
C14 Equity/total deposit 15.65 11.27 10.49
C15 Tot cap/tot dep 28.82 39.26 34.82
C16 Res loan loss/tot cap 5.56 4.14 5.07
D Liquidity ratio D1 Cash and sec/tot dep 40.81 44.08 23.12
D2 Deposits/tot asset 54.29 51.70 55.25
D3 Tot loans/deposits 124.68 121.68 134.04
D4 Res loan loss/loans 1.33 1.46 1.36
D5 Prov loan loss/loans 0.15 0.32 0.19
D6 Res loan loss/tot asset 0.87 0.84 0.97
D7 Demand dep/deposit 49.31 53.28 39.8
D8 Saving dep/dep 39.24 33.21 29.76
E Growth ratio E1 Net sales growth 16.55 22.6 20.71
E2 Oper inc growth 19.03 23.54 17.37
E3 Net inc growth 89.87 14.82 17.79
E4 Tot assets growth 60.56 38.71 16.98
E5 Equity growth 159.42 17.93 3.08
E6 EPS growth 89.7 15.47 15.32
E7 Div/share growth 13.04 9.82 14.89
E8 Bk value/shr growth 88.56 15.61 4.99
E9 Net margin growth 62.92 2 6.35 2 2.42 Table I.
E10 Loans growth 33.34 20.00 8.83 Performance indicators
E11 Renvstmt rate/shr 21.90 8.67 7.56 and original value for
(continued) three banks
BIJ Ratio Code Indicator CBA NAB WBC
13,1/2
E12 Renvstmt rate/tot 26.99 11.04 11.19
F Stock performance F1 P/E ratio close 9.52 11.84 14.36
F2 Price/book val close 2.00 2.07 2.64
F3 Price/cash flow 8.09 2 60.67 19.86
152 F4 Earning yield close 10.51 8.45 6.96
F5 Dividend yield close 4.69 4.82 4.24
F6 Tot investment return 20.54 19.2 40.59
F7 Dividend payout 44.67 57.09 60.81
F8 Mkt cap/equity 2.00 2.07 2.65
Notes: Description on the company codes: CBC stands for commonwealth bank; NAB stands for
National Australia Bank; WBC stands for Westpac Bank; Fiscal Year Ending of financial data is Year
2000/09/30
Table I. Source: All financial data obtained from World Database of Thomson Financial

values on the financial ratios of the three banks from Table II were inputted into the
program, and the program automatically grouped them into categories in accordance
to the amounts of the values, and then selected the values that were most significant.
This is shown in Table III.
All 59 financial ratios preliminarily selected were divided into 23 clusters via GRA,
and 23 ratios that represent the clusters were also retrieved (Table III). Among them,
A-IV, B-I, B-II, C-III, D-I, D-III, D-IV, E-IV and F-II have only one ratio each, which
indicates that the ratios have a lower degree of grey relation with other ratios, and
hence are independent from them.
Within A-I cluster (Table III), A1, A2 and A3 are represented by A2; within A-II
cluster, A4 and A9 are represented by A4; within A-III cluster, A5, A7 and A10 are
represented by A10; A-V cluster, A8, A11, A12 and A13 are represented by A8; within C-I
cluster, C1, C2, C3, C6, C7, C9, C10, C13 and C15 are represented by C7; within C-II cluster,
C4 and C8 are represented by C4; within C-IV cluster, C11, C12, C14 and C16 are
represented by C12; within D-II cluster, D2, D3, D4 and D6 are represented by D3; within
E-I cluster, E1, E2 and E7 are represented by E1; within E-II cluster, E3, E5, E6 and E8 are
represented by E8; within E-III cluster, E4, E10, E11 and E12 are represented by E10;
within F-I cluster, F1, F2, F6, F7 and F8 are represented by F7; within F-III cluster, F4
and F5 are represented by F4.

The evaluation result and analysis

The TOPSIS method was used to calculate the total performance score of each
bank. The calculation steps for TOPSIS are shown in appendix. After conducting
the TOPSIS, the authors found the ranking of the three banks in “profitability”,
“asset utilization”, “leverage”, “liquidity”, “growth”, “stock performance” as well as
“overall performance”, which are shown in Table IV.
We can see that Commonwealth Bank outperformed Westpac Bank and National
Australia Bank overall. However, using FSA (five-power analysis and stock
performance), we can see that the performance of Commonwealth Bank in “asset
utilization” was below the other two banks. The management of Commonwealth Bank
should be aware of this disadvantageous position and seek to improve. For National
Australia Bank, its overall performance ranked last. Yet, it has an edge in “liquidity”,
 Benchmarking
Indicators Code CBA NAB WBC
performance
A Profitability A1 0.825 0.439 0.357 indicators
A2 0.817 0.417 0.398
A3 0.813 0.415 0.408
A4 0.701 0.558 0.445
A5 0.625 0.582 0.519 153
A6 0.954 20.081 0.287
A7 0.537 0.646 0.542
A8 0.483 0.636 0.602
A9 0.720 0.518 0.463
A10 0.604 0.612 0.511
A11 0.483 0.598 0.639
A12 0.501 0.632 0.591
A13 0.500 0.636 0.587
B Asset utilization B1 0.506 0.697 0.508
B2 0.492 0.402 0.656
C Leverage C1 0.349 0.665 0.660
C2 0.244 0.696 0.675
C3 0.518 0.600 0.610
C4 0.998 0.060 0.017
C5 0.000 0.933 0.361
C6 0.478 0.625 0.617
C7 0.455 0.641 0.618
C8 0.880 20.044 0.473
C9 0.312 0.664 0.680
C10 0.422 0.636 0.647
C11 0.794 0.420 0.440
C12 0.719 0.493 0.490
C13 0.488 0.633 0.601
C14 0.713 0.513 0.478
C15 0.481 0.656 0.582
C16 0.647 0.482 0.590
D Liquidity D1 0.634 0.685 0.359
D2 0.583 0.555 0.593
D3 0.567 0.554 0.610
D4 0.555 0.609 0.567
D5 0.374 0.798 0.474
D6 0.561 0.542 0.626
D7 0.596 0.644 0.481
D8 0.661 0.559 0.501
E Growth E1 0.475 0.649 0.594
E2 0.545 0.675 0.498
E3 0.968 0.160 0.192
E4 0.820 0.524 0.230
E5 0.994 0.112 0.019
E6 0.972 0.168 0.166
E7 0.590 0.444 0.674
E8 0.983 0.173 0.055
E9 0.994 20.100 2 0.038 Table II.
E10 0.836 0.502 0.221 Vector normalization
E11 0.885 0.350 0.306 values on the financial
(continued) ratios of the three banks
BIJ Indicators Code CBA NAB WBC
13,1/2
E12 0.864 0.353 0.358
F Stock performance F1 0.455 0.566 0.687
F2 0.512 0.530 0.676
F3 0.126 20.943 0.309
154 F4 0.693 0.557 0.459
F5 0.590 0.606 0.533
F6 0.416 0.389 0.822
F7 0.472 0.603 0.643
F8 0.511 0.529 0.677
Table II. Notes: The values from Table I were input into equation (3) in appendix to produce these values

Cluster Represented by Ratios within the cluster

A A-I A2 A1, A2, A3

A-II A4 A4, A9
A-III A10 A5, A7, A10
A-IV A6 A6
A-V A8 A8, A11, A12, A13
B B-I B1 B1
B-II B2 B2
C C-I C7 C1, C2, C3, C6, C7, C9, C10, C13, C15
C-II C4 C4, C8
C-III C5 C5
C-IV C12 C11, C12, C14, C16
D D-I D1 D1
D-II D3 D2, D3, D4, D6
D-III D5 D5
D-IV D7 D7
D-V D8 D8
E E-I E1 E1, E2, E7
E-II E8 E3, E5, E6, E8
E-III E10 E4, E10, E11, E12
Table III. E-IV E9 E9
Financial ratios that F F-I F7 F1, F2, F6, F7, F8
represent their respective F-II F3 F3
cluster F-III F4 F4, F5

being ranked first. This shows that the overall performance was influenced by
“liquidity”. For its marketing strategy, it could strengthen its safety in deposits and
lending as well as the efficiency of its operations. In internal control, it could make
improvements in the liquidity of cash flow and cost control. For Westpac Bank, it is
time to exert proper safety measures in improving its financial position. They should
pay attention to the appropriate proportions between assets and owner equity, as well
as the use of financial leverage.
Comparing GRA to financial statement analysis results Benchmarking
In order to compare the results from using GRA, this paper used FSA to evaluate the performance
performance of the three banks. Within the banks, 59 financial ratios were used to
conduct a financial statement analysis. These financial ratios, as stated previously, indicators
include six dimensions: liquidity, asset utilization, leverage, profitability, growth and
stock performance. The analysis and result of FSA are shown in Table I.
From the result of FSA, the rankings of the three banks are the same as using GRA. 155
This result indicates that GRA has been successfully used in retrieving the
representative indicators. This result also indicates that GRA approach is better than
FSA approach in the empirical study, because GRA has retrieved the representative
ratios from original 59 ratios into 23 ratios. The reduced rate of performance indicators
by using GRA is ð59 2 23Þ=59 ¼ 61:02percent[3]. Table V shows the difference and
comparison between FSA and GRA.
Conclusions and future research
This paper introduced a new approach for performance evaluation, GRA. GRA is a
concept borrowed from the study of industry and increasingly applied to commerce.
This study also identified a ranking system that is applicable to the evaluation of
performance with small sample size where the distribution of data is unknown. Finally,
the study yielded expected results in an empirical study on three Australia’s major
banks. The conclusions are as follows:
.
To overcome the limitation of sample size and distribution type, GRA retrieved
financial ratios that could represent the cluster to which they belonged. This
could avoid the waste of resources due to the uncertainty of relations among the
ratios when using them for analysis.
.
TOPSIS has been used for the ranking of the subject of the study. By considering
the performance of the banks relative to each other, this method can assist bank
managers in understanding the current situation of their companies, so that they
can plan action for improving performance.
.
Comparing the GRA results with FSA, the result indicates that GRA approach is
better than FSA approach in the empirical study, because GRA has retrieved the
representative ratios from original 59 ratios into 23 ratios. The reduced rate of
performance indicators by using GRA is ð59 2 23Þ=59 ¼ 61:02percent:
The major contribution of this paper is the use of GRA methodology to retrieve ratios most
commonly used in financial analysis to tackle the problems of sample size and distribution

Item Ranked first Ranked second Ranked third

Profitability (A) CBA (0.882) WBC (0.317) NAB (0.162)

Asset utilization (B) WBC (0.573) NAB (0.429) CBA (0.263)
Leverage (C) CBA (0.514) NAB (0.497) WBC (0.255)
Liquidity (D) NAB (0.561) CBA (0.440) WBC (0.183)
Growth (E) CBA (0.900) NAB (0.281) WBC (0.081)
Stock performance (F) WBC (0.844) CBA (0.814) NAB (0.115) Table IV.
Total CBA (0.686) WBC (0.410) NAB (0.334) The ranking of overall
performance of the three
Note: The numbers in parentheses indicate the proximity to the positive ideal solution banks
BIJ
The ranking in the
13,1/2 empirical study Indicators
Method Basic principle Strengths Weaknesses (Yr of study) used

FSA People use this Objective – It is There is no Commonwealth 59

method with the the reflection of criterion for bank ranks first, financial
156 belief that the actual events selecting a ratio with second, ratios
result of business Concrete – All that is agreeable Westpac bank and
activities of the data in the by all users National Australia
firm would be financial The figures in the bank third (2000)
reflected in its statement could be financial
financial quantitified statement have
statement Measurable – been added or
Since the data in simplified, and
the financial could not satisfy
statement could be the needs of all
quantitified, they users
are measurable Financial
statement could
not express
qualitative
information, such
as ability, morale,
potential and trust
GRA Based on the No rigid Cannot directly Commonwealth 23
homogeneity or requirement in handle qualitative Bank ranks first, financial
heterogeneity of sample size issues with second, ratios
the trend Can be applied (non-quantifiable) Westpac Bank and
development of when the The criteria for National Australia
elements to find distribution of choosing grey Bank third (2000)
out if there is a data is uncertain relation coefficient
grey relation Is based on data value directly
between two analysis, and is affects the final
indicators and to free from evaluation result
what extent traditional
subjectivity in
decision-making
The method of
Table V. calculation is
A comparison of FSA simple and easy to
versus GRA apply

uncertainty. This could avoid the waste of resources due to the uncertainty of relations
among the ratios when using them for analysis. From the empirical study done in this
paper, the result indicates that GRA approach is better than FSA approach, because GRA
has retrieved the representative ratios from original 59 ratios into 23 ratios.
When the results of GRA are compared with those of FSA, the same outcome
findings emerge. Having the same result is interesting and meaningful, particularly
with respect to the reduced rate of performance indicators by using GRA. GRA has
retrieved the representative ratios from original 59 ratios into 23 ratios. As Table III
shows, 59 ratios can be divided into 23 clusters and can be seen that any one of the six
categories of indicators can be replaced by another or can stand independent of Benchmarking
another. Future research could continue to explore or discuss the relationship between performance
these indicators. For example, within A-III cluster, A5, A7 and A10 (as Table III shows)
are represented by A10 in the empirical study. If so, is A10 strictly speaking providing indicators
us with the same information as A5, A7 and A10? In other words, there are some ratios
are redundant within A-III cluster. If so, it would be a meaningful task to find these
ratios out in the future. 157
This paper conducted a review of literature and five-power analysis to aggregate
financial ratios appropriate for the analysis. This method may result in incompleteness
in the aggregation of ratios, and requires adjustment when other issues for analysis are
involved. Future research could set up a specific model for the preliminary selection of
financial ratios with a new to make studies of this kind more complete.

Notes
1. Five-power analysis in the study is a combination of liquidity, asset utilization, leverage,
profitability, and growth.
2. We know that there are big four banks in Australia. Unfortunately, the World Database of
Thomson Financial does not provide complete financial data of ANZ Banking Group.
Therefore, we restrict our analysis in Australia’s three major banks.
3. The performance indicators set originally contains 59 ratios for FSA. GRA has retrieved the
representative ratios from original 59 ratios into 23 ratios. So the reduced rate of
performance indicators by using GRA is ð59 2 23Þ=59 ¼ 61:02percent

References
Collins, R.A. (1980), “An empirical comparison of bankruptcy prediction model”, Financial
Management, Vol. 9 No. 2, pp. 52-7.
Deng, J. (1982), “Control problems of grey system”, System and Control Letters, Vol. 1 No. 5,
pp. 288-94.
Deng, J. (1989), “Introduction to grey system theory”, The Journal of Grey System, Vol. 191,
pp. 1-24.
Espahbodi, P. (1991), “Identification of problem banks and binary choice models”, Journal of
Banking & Finance, Vol. 15, pp. 53-71.
Fielding, G.J., Babitsky, T.T. and Brenner, M.E. (1985), “Performance evaluation for bus transit”,
Transportation Research, Vol. 19A No. 1, pp. 73-82.
Grosskopf, S. and Valdmanis, V. (1987), “Measuring hospital performance: a nonparametric
approach”, Journal of Health Economics, Vol. 6, pp. 89-107.
Huang, M-C. (1986), A Study of Domestic Bank’s Performance, Basic Finance Publication, Taipei.
Hwang, C.L. and Yoon, K. (1981), Multiple Attributes Decision Making: Method and Application,
Springer, New York, NY.
Lin, C. (1998), “A study of efficiency evaluation in Taiwan’s securities dealers”, Securities
Finance Quarterly, Vol. 58, pp. 1-24.
Lin, J-G. (2000), “Performance evaluation of restaurant service quality in airport”, unpublished
Master thesis, Department of IE and Management, Chung Hua University.
Maisel, L.S. (1992), “Performance measurement: the balanced scorecard approach”, Journal of
Cost Management, Vol. 2 No. 2, pp. 47-52.
BIJ Pantalone, C.C. and Platt, M.B. (1987), “Predicting commercial bank failure since deregulation”,
New England Economic Review, Vol. 13, pp. 37-47.
13,1/2 Seiford, L.M. and Zhu, J. (1999), “Profitability and marketability of the top 55 US commercial
banks”, Management Science, Vol. 45, pp. 1270-88.
Shih, Y-F. (2000), “External performance evaluation of distribution centres in logistics: a fuzzy
theory approach”, unpublished Master thesis, Department of International Business, Ming
158 Chuan University.
Shih, K-C., Wu, G-Q. and Huang, Y-P. (1994), The Relation of Grey Information, Chyuan-Chen
Technology Publishing, Taipei.
Tapas, K.D. (1998), “Benchmarking: theory and practice”, IIE Transaction, Vol. 30 No. 9,
pp. 861-2.
Tsai, D. (2000), “The performance evaluation model of the Commercial Bank of Taiwan: by grey
relation analysis and factor analysis”, unpublished Master thesis, Department of
Management Science, Ming Chuan University.

Appendix. Calculation steps of TOPSIS

Technique for order preference by similarity to ideal solution (TOPSIS) was proposed by Hwang
and Yoon (1981). The purpose is to find a solution closest to the “positive ideal solution” and
furthest from the “negative ideal solution”. “Positive ideal solution” refers to the most effective or
least costly value among a set of feasible solutions. Conversely, a value of least effectiveness and
highest cost would be the negative ideal solution. TOPSIS is used as the ranking method with
GRA in the empirical study. The advantages of this method are that it is relatively simple and it
yields a highly reliable preference order. The steps are as follows:
Step I – Normalization of initial value. In this study, the application of GRA and TOPSIS
uses vector normalization, which uses the ratio of the original value and the square root of
the sum of the original indicator values. The formula is as follows:
xij
r ij ¼ sffiffiffiffiffiffiffiffiffiffiffiffi ðA1Þ
X m
x2ij
i¼1

where i is the ith bank, j is the jth financial ratio, rij is the performance value of financial
ratios after vector normalization for magnitude and direction, xij is the original
performance value of financial ratios and, m is the number of bank companies.
Step II – Find the positive ideal solution (Aþ ) and negative ideal solution (A2 )
    
þ 0
A ¼ max xij j j [ J ; min xij j j [ J ji ¼ 1; 2; . . . ; m
i i

n o
¼ Aþ þ þ þ
1 ; A2 ; . . . ; Aj ; . . . ; Ak

    
A2 ¼ min xij j j [ J ; max xij j j [ J 0 ji ¼ 1; 2; . . . ; m
i i

n o
¼ A2 2 2 2
1 ; A2 ; . . . ; Aj ; . . . ; Ak

J ¼ {j ¼ 1; 2; . . . ; kjk is efficiency}; J 0 ¼ { j ¼ 1; 2; . . . ; kjk is cost}

 Here, efficiency criteria imply a larger indicator value and a higher performance score; Benchmarking
Cost criteria imply a smaller indicator value and a higher performance score.
Step III – Calculate the distance from each solution (bank) to the positive ideal solution
performance
ðS þ
i Þ and to the negative ideal solution ðS i Þ
2 indicators
vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
u k  2
uX
Si ¼ t
þ
vij 2 Aþ i ¼ 1; 2; . . . ; m
j¼1
j
159
vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
u k  2
uX
S2
i ¼t vij 2 A2 j i ¼ 1; 2; . . . ; m
j¼1

Sþ
i is the shortest distance from the ideal solution (bank); S 2
i is the farthest distance from
the worst solution (bank).
*
Step IV – Calculate the proximity of each solution (bank) to the ideal solution ðC i Þ: It is
defined as

* S2
i
ci ¼ ðA2Þ
Sþ
i þ S2
i
*
The step is to calculate the relative closeness to the ideal solution ðC i Þ:
*
Step V – Conduct the ranking among solutions (banks). Based on the value of C i from
Step IV, rank the performance among the solutions.

For TOPSIS, the chosen indicator should have the shortest distance from the ideal
solution and the farthest from the worst. The ideal solution is the one that enjoys the
largest efficiency indicator and the smallest cost indicator among each of the
substitutive bank companies. The worst solution is the one that enjoys the smallest
efficiency indicator and the largest cost indicator among each of the substitutive bank
companies.

Corresponding author
Chien-Ta Ho can be contacted at: bruceho@mail.fit.edu.tw

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

BIJ
13,1/2 SWOT analysis for Air China
performance and its experience
with quality
160
A.M. Ahmed
The European Centre for Total Quality Management, University of Bradford
School of Management, Bradford, UK
M. Zairi
e-TQM College, Dubai, UAE, and
K.S. Almarri
The European Centre for Total Quality Management,
University of Bradford School of Management, Bradford, UK

Abstract
Purpose – To report on the lessons learned during the implementation of TQM principles in Air
China as a response to the dramatic changes in both international and domestic markets.
Design/methodology/approach – This research work is based on analysing secondary case
studies in the airline industry to identify best practice and critical success factors of total quality
management implementation. The use of SWOT analysis was selected in the case of Air China.
Findings – Air China is the largest air carrier in China in terms of traffic volume and company
assets. It own most updated fleet and competent repairs and maintenances expertise. It owns good
reputation in both international and domestic market, quality service and keeps the number of loyal
frequent flyers rapidly increasing. Distracted by domestic operations in term of resources,
organizational concentrations and management time, the international business get to be mediocre.
China airline industry is growing faster than GDP increase, and the trend will continue in next five
years. Air China faces imminent aggressive competition from world leading airlines and price wars
triggered by domestic player, when Civil Aviation Administration of China has to deregulate under
requirement of WTO agreements.
Originality/value – This paper is a case study reporting of the strengths, weaknesses, opportunities
and threat analysis technique. The paper provides empirical data to identify those factors that play
key role in implementing TQM successfully within the Airlines Industry and in particular Air China.
Keywords Total quality management, SWOT analysis, Business performance, Customer satisfaction,
Airlines, China
Paper type Case study

Benchmarking: An International
Journal
Vol. 13 No. 1/2, 2006
pp. 160-173
q Emerald Group Publishing Limited
1463-5771
DOI 10.1108/14635770610644655
Introduction
The airline industry has undergone significant restructuring in recent years. Airlines,
formerly rivals in a highly regulated industry, have become opportunistic seekers of
co-operation. In today’s world, mega-carriers and small airlines are working together
rather than competing with one another. Forms of co-operation include sub-contracting,
code sharing, franchising and the formation of global marketing networks. Such
alliances allow firms to focus on their respective core competencies, while drawing the
benefits of scale economies. In essence, co-operation among competitors has led to
increased competitiveness. This has accelerated the trend of joint marketing, and the
airline has become characterised by the desire to belong to a global network. The SWOT analysis
tendency has been to strive for a global presence. for Air China
The transportation industry was traditionally among the most regulated and
protected sectors of the economy. For many national flag-carriers, neither competition
nor market forces were considerations. Governments set prices, routes, services, etc.
More recently, industry structure has changed, reflecting a new operating environment
and giving rise to opportunities for joint-marketing alliances. During the past two 161
decades, competitive forces have replaced the government as instruments of regulation.
Previously protected airlines found themselves competing in a new environment.
Whereas airlines used to operate independently of one another, the industry structure
has been shifting, as there has been pressure to collaborate with competitors, in order to
become competitive.
As early as 1914, passengers on Zeppelin airships were served champagne with their
in-flight meals. During the 1920s, flying dining rooms were introduced, and chefs
prepared hot meals aboard the Zeppelins. Airlines introduced railway-style, uniformed
stewards and restaurant-style tables adorned with vases of flowers and silver cutlery. In
1928, Pan Am became the first US airline to employ cabin attendants to serve meals. Also
in 1928, Western Airlines (eventually absorbed in Delta Airlines) served meals brought
to their aeroplanes, by Cadillac, from a restaurant near Los Angeles Airport. In 1929, Pan
Am introduced in-flight movies on its flights between Miami and Havana (Dana, 1999).
In 1930, United Air Lines introduced the world’s first stewardess. Ellen Church was
the first woman to serve a meal on an aeroplane. Pan Am preferred to employ stewards
who had served as professional waiters or pursers on ships. In 1934, Pan Am
inaugurated the S-42 on its route from Miami to Buenos Aires. The airline set a
precedent by installing facilities for heating food in-flight on long over-water journeys.
Stewards would take orders from passengers and radio ahead for meals, which were
served to passengers in a special dining area of the aeroplane (Dana, 1999).
Dinner in flight became a formal affair, often in tuxedos and evening gowns.
Passengers enjoyed a feast, cooked aboard the aeroplane, and served on linen-covered
tables in a fine dining salon set with china, crystal and fresh flowers. During the 1940s
and 1950s, passengers on the Boeing 377 Stratocruiser could get staterooms with
private washstands, berths, divans, clothes closets and a reclining lounge chair.
Dinner, served downstairs, was the highlight of the flight (Dana, 1999).
In 1956, when tourist class was introduced, the International Air Transport
Association defined what constituted a sandwich and dictated what airline meals could
include. Regulations stipulated exactly what could be served and permitted quantities.
In 1958, Pan Am inaugurated the Boeing 707, popularising jet service. Flying at high
altitudes, jets introduced new challenges. New jets provided less space to prepare
food, and less space to eat. Jet travel brought about simpler and less tasty meals
(Dana, 1999).
During the late-1960s, Miami-based Eastern Airlines introduced “Famous
Restaurant Flights”, serving meals on Rosenthal China. Northeast Airlines – its
principal competitor on the US East Coast – responded with steak and champagne in
coach as well as in first. Double-decker airline service was re-introduced by Boeing 747
in 1969. Air Canada used the upstairs cabin as a bar for first class passengers. Pan Am
used the upper deck as a restaurant with individual place names. Downstairs, British
Airways installed a microwave (Dana, 1999).
BIJ Air China case study
Air China was founded on the 1st of July 1955. Its headquarter is based in Beijing. It
13,1/2 engages in international and domestic passengers and cargo flight services. To unify
its facility image and simplify its repairs and maintenances, currently, it fleet of total
118 aircraft exclusively is consisted of Boeing models, including 56 B747s, 17 B777s,
19 B767s and 26 B737s. with over than 28,600 of seat capacities. It has established
162 hub-spoke style passenger and cargo transport network. The hub of this network is
Beijing International Airport.
The company is operating 339 routes, which consists of 53 international and 286
domestic, operating more than 1,000 scheduled flights on weekly basis, serving 29
cities in 19 countries. About 66 per cent of its revenue was sourced from the domestic
market. Over the last two years, Air China has experienced dramatic changes in both
international and domestic market.

Growing domestic market

Fierce competition and sluggish market demand has forced the compant to turn its
business focus on the fast growing domestic market, where it enjoys the protections
from the Civil Aviation Administration of China (CAAC). Chinese Government has
adopted an aggressive fiscal policy by lowering interest rate to 2 per cent, aiming to
boast GDP growth. In 2001, China’s GDP reached 7.3 per cent, which is considered to
be the fastest growing economy compared with both the main developed and
developing economies worldwide. International Monetary Federal forecasted that
China economy would continue to grow with the rate of 7 per cent per year for the
period 2002-2007. The transportation industry especially the airline sector will lead this
growth. According to the CAAC predicting indicators, the airline sector will grow at a
rate of 10-15 per cent per annum. There are other factors that contribute to the growing
demand of the domestic market for air service in China. These to include: opening up
and reform the country’s development strategy, boasting the tourism industry and
attracting more people to invest or visit China. Tables I-III illustrates statistics of this
growth.
In April 2001 and under the regulations of the CAAC, 56 national and local airlines
are categorised into three groups as shown in Tables IV and V. Air China operated in
developed areas such as America, Japan, Western Europe and Austria since it found,
the domestic operations grew rapidly after the acquire regional airlines; the other two

Year 2001 Proportion Year 2000 Growth

Regions 0.000 person per cent 0.000 person per cent

Japan 2.391 21.3 2.190 8.4

Southeast Asia 1.891 16.2 1.631 10.3
Korea 1.684 15.0 1.266 24.8
Russia 1.201 10.7 1.073 10.7
Western Europe 1.156 10.3 1.075 7.0
USA 0.954 8.5 0.989 5.9
Other areas 2.021 18.0 1.756 15.0
Total 11.226 100.0 9.890 13.5
Table I.
Visitors to regions Source: Annual Report of Chinese Tourism Administration (2001)
groups aggressively expanded into international market while mainly focused on SWOT analysis
domestic business. for Air China
While the factors like price and schedule arrangement are well below the average
level, the choices basing on brand image and service are largely exceed the other
two, Air China positions itself with quality service while other two groups competed

163
Business and meetings 1,985 17.7
Visiting friend and family 4,923 43.9
Sightseeing and leisure 393 3.5
Work and crew 1,289 11.5
Others 2,636 23.5
Total 11,226 100.0
Table II.
Source: Annual Report of Chinese Tourism Administration (2001) Purpose of visits

By sea 1.488 13.3

By air 5.999 53.4
By railway 0.446 4
Motor vehicles 1.762 15.7
Others 1.531 13.6
Total 11.226 100.0
Table III.
Source: Annual Report of Chinese Tourism Administration (2001) Transportation

Item Air China group Eastern Airline group Western Airlines group

Hub Airport Beijing Shanghai Guangzhou

Capital £4.3 billions £3.64 billions £3.85 billions
Planes 118 118 180
Routes 339 437 606
Domestic routes 286 383 512
International routes 53 54 94
Employees 20,235 25,109 34,089 Table IV.
National and local
Source: www.CAAC.com.cn airlines groups

Airlines Brand image Price Plane model Schedules Service FFP

Air China 52.91 3.88 8.55 19.33 10.72 4.62

Eastern Airlines 30.14 7.31 3.65 40.79 8.83 9.28
Southern Airlines 24.39 7.32 6.5 47.15 4.07 10.57 Table V.
Average 35.81 6.17 6.23 35.76 7.87 8.16 Comparison of passenger
choice criterion ( per cent)
Source: CACC 2000-2001 Market Survey between the three airlines
BIJ with lower price and completed schedule arrangements. Recently, all three
13,1/2 groups are experiencing deteriorating performances due to other global factors,
which will be discussed in the next section. For example, profitability was
unacceptably low as 0-3 per cent, and passenger load factor, a critical operation
indicator (utilisation of fleet capacities) was below 60 per cent, contrasting with
Singapore Airlines’ 81.6 per cent, BA’ 76.4 per cent and worldwide 71 per cent in
164 2001. Under the pressures, revenue maximising and cost efficiency are becoming the
first considerations when they stipulated business strategies. The details are listed
in following Table VI.

International market
The demand for air travel has grown every year since the Second World War with the
exceptions of the Gulf War in 1991 and 2001. Since 2001, there are an emergent factors
curbed and add to the uncertainty of the airlines industry’s growth. These factors are
summarised as follows:
.
Worldwide top three: American, Japan and EU economies are further stuck into
recession.
. September the 11 attacks heavily blow the world economy, dragging down the
international airline industry.
.
USA probable military actions against Iraq posing additional threat to the
airline industry.

Though these events shall keep blocking the growth, the speedy pace of economic
globalization and increasing demand of leisure travel by prosperous consumers kept
the air traffic growth of approximate 3.5 per cent per annum according with estimation
by the International Civil Aviation Organisation. But main players in the global market
took pessimistic stances by cutting down the capacity, furiously seeking consolidation
within the industry, and lobbying the regulatory authorities for opening up the skies.
As demonstrated in Table VII, in 2001, there were approximately 75.3 million
passengers uplifted by China airlines, equivalent to 2.3 per cent of world air travel
demand measured in terms of passengers, growth rate is fastest among other
developing and developed regions.
China geographically is similar with USA, the overall turnover represent 7 per cent
of that of USA; and in term of passengers uplifted in 2001 China airlines was similar

Air China Eastern Airlines Southern Airlines

Operation performance (per cent) (per cent) (per cent) Remark

Overall load factor 59 55 54 ¼ RTK/ATK

Passenger load factor 63 60 60 ¼ RPK/ASK
Domestic 62 61 60
International 64 59 61
State owned shares 100 61 82
Net profitability 2.4 0.13 3.3 ¼ Net profit/revenue
Table VI.
Comparison of the three Notes: RTK: revenue tone kilometers; ATK: available tone kilometers; RPK: revenue passengers
groups operating kilometers; ASK: available seat kilometers
performance Source: Annual reports for the above three groups (2001)
 SWOT analysis
Region Total passengers 0.000.000 personsa Change-per cent
for Air China
Africa 77.3 2 1.70
Asia/Pacific 554.6 2.00
China 75.3 11.50
Europe 993.5 2 0.60
UK 71.1 0.10 165
Latin America/Caribbean 166.1 1.20
Middle East 66.9 2 0.80
North America 1,353.80 2 6.30
Total 3,212.20 2 2.60
Note: aTotal passengers enplaned and deplaned, passengers in transit counted once Table VII.
Source: Airports Council International Report (2001-2002) World air travel demand

with UK’s, but the structure was totally different. For example, China airlines
operations are mainly confined within domestic market, more than 90 per cent traffic
from domestic market, contrasting with British airlines 25 per cent (Airports Council
International, 2001). In international competition arenas, though distances exist, Air
China is pacing with world leading airlines, endeavouring to provide quality services
(Table VIII).

Research methodology
This research work is based on analysing secondary case studies in the airline industry
to identify best practice and critical success factors of total quality management
implementation. Therefore, the use of SWOT analysis was selected in the case of Air
China. The SWOT analysis is the process of analysing organisations and their
environments based on their strengths, weaknesses, opportunities and threats. This
includes the environmental analysis, the process of scanning the business environment
for threats and opportunities, which is considered as external factors, and the
organisational analysis, the process of analysing a firm’s strengths and weaknesses as
internal factors (Weihrich, 1982).
SWOT analysis was carried out for Air China and the results are summarised as
follows:

Strengths
Air China is the largest air carrier in China in terms of traffic volume and company
assets. Over 20,000 employees are working for Air China, including more than
2,300 pilots and 4,520 flight attendants. It enjoys well-trained flight crew who are
experienced in international operations and services. It own most updated fleet and
competent repairs & maintenances expertise. Its information systems are the most
advanced among Chinese airlines and compatible with its operation and service. It
owns good reputation in both international and domestic market, quality service and
keeps the number of loyal frequent flyers rapidly increasing. Expanding global
alliances (Table IX) provide passengers with more route/schedule choices and
completed travel package services.
BIJ
Services items British Airway Singapore Airline Virgin Airlines Air China
13,1/2
First class 5 5 n/a 3
Business class 4 4 4 3
Premium economy class 3 n/a 4 3
Economy class 3 3 3 3
166 Airport services 4 3 3 3
Airport lounges 4 3 5 3
Onboard-general
Safety procedures 4 4 4 4
Inflight entertainment 4 5 5 2
Comfort amenities 3 4 4 3
Reading materials 3 3 4 3
Seating-long haul
First class 5 5 n/a 4
Business class 5 4 4 4
Premium economy class 3 n/a 4 3
Economy class 2 3 3 3
Seating-short haul
Business class 4 4 n/a 4
Economy class 3 3 n/a 3
On board catering-long haul
First class 4 5 n/a 4
Business class 3 4 4 4
Premium economy class 3 n/a 3 3
Economy class 3 3 3 3
On board catering-short haul
Business class 3 4 n/a 4
Economy class 3 4 n/a 2
Cabin staff service
First class 4 5 n/a 4
Table VIII. Business class 3 4 4 3
Quality services provided Economy class 3 4 3 3
by airlines Overall 4 5 4 3

Air China flights Operating airline

CA3100-CA3999 Shanghai airlines
CA4000-CA4999 China Southwest airlines
CA5000-CA5999 CNAC Zhejiang airlines
CA6000-CA6099 Lufthansa airlines
CA9020/21/28/29 Lufthansa airlines
CA165/6 Austria airlines
CA187/8 TAROM
CA191/2 SWISSAIR
CA195/6 SAS
CA121/2 FINNAIR
CA981/82/89/90 Northwest airlines
Table IX. CA7000-CA8999 US partners (Northwest airlines, Continental
Air China global alliances airlines,West airlines)
Weaknesses SWOT analysis
Air China is operating cross board international and domestic market competing with
world leading giant airlines as well as local small operators. This lacks clarity on the
for Air China
strategic direction largely dilute its capabilities and seriously confused its brand in
markets. This has been reflected on its low profitability and utilisation of capacity
(load factors). Many quality services initiatives and practices are easily coped by the
competitors. Distracted by domestic operations in term of resources, organizational 167
concentrations and management time, the international business get to be mediocre.

Opportunities
China airline industry is growing faster than GDP increase, and the trend will continue
in next five years. Worldwide deregulations make the skies more accessible; the route
agreement is easier to be achieved. The number of foreign visitors and investors to
China increase fast. Complementary industry like tourism will increase demand for
airline service. CAAC strong regulation and protection render with chances for
consolidation and optimisation. Customers are getting to be richer, tend to be less price
conscious and prefer to choose quality service.

Threats
Air China faces imminent aggressive competition from world leading airlines and price
wars triggered by domestic player, when CAAC has to deregulate under requirement
of WTO agreements. Foreign airlines will access to Chinese computerized seat ordering
system; this provides Chinese passenger with more choices. Trains and buses
businesses dramatically improve speed and services in their medium/long distant
routes, attracting passengers away from air service. Most of passengers complain the
ticket price as unacceptable, premium is not value for money, they are pressurising
CAAC to cut price down. Incapability of industry infrastructure like industry-wide
yield management system (YMS), global distribution systems (GDS) undercut its
management capability.

The way forward to achieve excellence 2008

The company defined business activities from the passenger points of view.
Passengers travel process was divided into six phases as shown in Figure 1, starting
from passenger enquiry and ending up with passengers being out of operations.
Around passenger travel process and satisfaction drivers, the company identified key
business activities:
.
on/off flight service;
.
people management;
.
supplier network management;
.
information technology adoption and integration;
.
passenger perception management; and
.
customer satisfactions measuring and complaint handling.

On/off flight service

Based on Kano model (Figure 1), Air China has classified customer satisfaction drivers
according with the degree of tailoring services to individual passenger. The basic
BIJ
13,1/2

168

Figure 1.
Key business activities
in the passenger
travel process

needs that the company has to deal with are safety, getting the luggage to the right
place, and punctuality. The services must meet the basic requirement first before
climbing the into higher satisfactions stage. These activities have, however, been in
focus over a long period of time.
In order to become more competitive, the services must focus on qualifier
drivers, i.e. providing more available choices on food menus, or enhance seating
comfortability. Just meeting customer requirements is not enough; delighting
passengers is only the way to differentiate business from other competitors.
For example, Domestic Chartered Flight from Huhohort City, Inner Mongolia
capital, to Beijing created the Mongolian style service package: stewardesses cloth with
Mongolian costumes, cabins is full of Mongolian traits: music, videos, food, beverages,
and interior decorations. Holidaymaking passengers were fascinated with this exotic
atmosphere. Bearing rationales of Kano model in mind, the company prioritises the
activities and deploys human and technological resources accordingly.
As for people-technology mix, the company insisted that an employee is more
important than technical system. The creed is “people serve people”. Applying this in
practical way, a new purpose was initiated to integrate the menu planning, processing
with first/business ticket ordering and check-in system. The idea is that when
passengers order the ticket, they can order lunch/dinner, chefs would be informed to
prepare food plan. When information is confirmed upon check in, food is to be processed
and lifted on board. In this way, passenger taste can be accurately matched with.
During experimental stage, the passenger and ticketing staff stared at the complex
menus in the computer screen and were confused, they have no idea about what they
want. Later days, this nervousness was mitigated by colourful menu printings being
placed in the counter, passengers then choose food, and staff input their choices in the
system. Simplified! But when passengers are on board, their ideas often change when
they see other choices. At that moment, they have no other choices but stick to their
original ones. This initiative embarrassed staff and added hardness to passengers.
So management immediately postponed the plan. Subsequently, members from
cabin crew, food ordering and technical functions formed a menu improvement team.
They adopted Deming PDCA (plan, do, check, act) approach to develop Flexible Menu SWOT analysis
System for first/business classes. In terms of plan, they set up initial database based on for Air China
the past food records on different passengers’ preferences, then they input new changes
in passenger taste and make the in-flight menu to meet the updated needs (do). This
followed by analysing the feedback from voyage reports and passengers (check), then
update database consequently (action).
169
People management
When dealing with people, main principles of Air China as shown in Figure 2 can be
described as:
.
Excellence relies on people. Programme strategies would not be improved and
fulfilled without people participation, enthusiasm and contribution. People
satisfaction is premise to achieve customer satisfaction.
.
Managerial effectiveness. Its role can be defined as facilitator to build up
partnership between the individual and the organization, and between the
organization and its customers.
.
Organizational productivity. Quality should be seen to start from the top. The
leadership of the organization must be intimately involved to see that the quality
paradigm is planted into the minds and hearts of all staff.
.
The nonhuman side of the organization. Such equipment, facilities, processes and
systems only can alive in hand of people with enthusiasms, so-called
advancements or perfects are meaningless and burdens for company unless
they can contribute to both employee and customer satisfaction.

Figure 2.
People management in
Air China excellence
2008 programme
BIJ Among kinds of change initiatives, leadership, teamwork, and training are top
13,1/2 priorities in the initial stages of the programme. Senior managers made conscious
efforts to reinforce the customer-driven focus and quality-orientation conveyed in
vision; emphasised the program success is totally depends on people, no technologies
can work as replacement; and clarified objectives is to make all people satisfaction. All
these messages communicated through kinds of channels such as company newsletter
170 (Flying Phoenix), meetings, e-mail, more often in every informal occasions.
Team building is a key to successful leadership behaviour in a TQM organisation
(Puffer and McCarthy, 1996). Air China management required the company should be
organised around processes, and be run by cross-functional teams. They worked in
teams, building role model for whole company. They not only encourage teamwork
style, but also provide facilitating supports: defining processes, providing
methodologies of team formulation, and empowering to teams. Cross-functional
teams are bringing together people with different expertise. Such teams share
responsibility and credit. Teamwork enhanced service quality in two examples. First,
check in staffs managed the queue by “seeing”, the available staff moves to busy
counter when he/she sees the busy scene and queue. Only focus is clearing off queue
before all the counters. Second, cabin crew of the flight from Beijing to Frankfurt
generates new food service idea on the international routes: Western Format, Chinese
Content. That means serving Chinese food with European styled containers and
serving manners instead of previous pure Chinese or Western food choices. The value
equation is fantastic food plus familiar, therefore, comfortable services. Management
rolled out the idea to all European routes now.

Supplier network with partnership

Under the new quality strategy, the company decided to eliminate the short-term
suppliers as possible; the reconfiguration of supplier network aimed to build up
long-term partnership, which contribute to quality enhancement. The rationale of
suppliers partnership is demonstrated in the Boeing Commercial Aircraft Company
has been selected by Air China as the partner of supplying the most critical purchased
item-aircraft as the part of new quality strategy the company resold the other model to
simplify the repairs and maintenances (R&M), and to unify facilities image. Now its
fleet exclusively consisted with a series of Boeing models. The company assigns its
employees to work with Boeing on various aspects of the aircraft related to
modification of cabin classes such as seating redeploying, in-flight entertainment and
communication facilities to serve passengers. General agreements are reached for
upgrading technology, designing and recommending new model suitable for quality
service, Figure 3.
To scrutinize suppliers with new quality standards, the auditing group is formed,
consisting of people from on board and ground service, quality, purchasing, costing,
logistics, and long-term strategically important suppliers. Taking 6 months, they
selected 156 suppliers to sign long-term agreements, cutting down from previous 503.
The number will be halved in next few years.
Most significant selection criteria is if supplier can economically (cost/quality) and
socially (parternership/participation) recognized and served the new quality strategies.
For example, one of famous food brand company previously changed the designs of
Pasta sauce twice annually, supplying to business classes of flights to Mediterranean
 SWOT analysis
for Air China

171

Figure 3.
Supply network
partnership contributes
to service quality
Source: Operation Strategy: Partnership Supply (N.Slack & M.Lewis 2002)

areas, and when required to add change frequencies to 4 times one year, this company
rejected this requirement, so it was eliminated from the supplier list.

Information technology adoption and integration

As mentioned in the above sections, Under Excellence 2008 programmme, the
technology is classified as nonhuman side of the organisation, and management was
very vigilant on risk of dehumanization. The adoption of information technologies
serves as the fabricating the different phases of passengers travel process as shown in
Figure 4 and binds them together it should enable both passenger and employee to
control over travel/service process to facilitate friendly interactions.
Information technology is now being used substantially in areas like passenger
enquiry, ticket ordering, luggage tracing, check-in, frequent flyer programme (FFP),
passenger information and signage system, on board services. Through choice of
technology, Air China can develop processes and procedures as simple and
understandable as possible. The belief is if the service activities are made simpler for
passengers to understand, learn, and complete, passengers would feel more confident
to control his/her own travel.
For example, Air China computerised its FFP, so as to member of Friend club can
easy to exchange his miles record with items like class upgrading, price discounting,
free hotel accommodation, and gifts. This system also integrated with alliances to

Figure 4.
Principles for choice of
information technology
in Air China
BIJ realize exchangeability of loyalty. Company called this program with “Flying Miles
13,1/2 Banking”. The number of club member increased dramatically from 37,000 in 1999 to
600,000 in 2002. A passenger describes this as “easily understanding, and really
exciting.” The company invested to install GDS. Passenger can access it through
internet everywhere, and get immediate and comprehensive information relevant to
their travel, such as flight schedules, services choices, luggage weight tolerance, map of
172 routes, policies and procedures, destination area information on hotel, car renting,
travel agency, weather conditions, shopping, festivals, etc. Taking views from
passengers’ eyes, the system keeps updating, therefore, the passengers can manage
their travel easier than before.

Passenger perception management

There are four main considerations when Air China stipulate the passenger perception
management strategy:
(1) What are the passengers’ choice criterions?
(2) How passengers view the company’s performance on these criterions?
(3) Defining strategies: targeting, maintaining, and improving.
(4) Managing the perception through flexible pricing.

Taking an approach similar with the model of achieving differential service quality,
the company developed the strategies to manage the passenger perceptions. From the
average results in Table IX, the criterions can be prioritised as:
.
brand image;
.
schedules arrangement;
.
FFP;
.
on/off board services;
.
plane model; and
.
price.

Conclusions
Air China has tried to build strategy on genuine understanding of the customers’ true
needs. It identified key customer satisfaction drivers, then turned into a foundation to
shape company development strategies and innovations. The objective is to realize the
quality-strategy integration.
This integration is seen throughout the process of strategy management. Leaders
vision for the future changes in competition and customer requirement and clarify the
mission with 5-10 years span of time, correspondently stipulate business goals. Based
on preliminary market research and customer analysis, multi-teams generated
strategic ideas, e.g. defining customer satisfaction drivers and specifying criteria in
main four service areas; Formulating strategies choices for management to make
strategic decisions. The next phase in the process began with defining the key business
activities around passengers travel process, developing new services, and deploying
policy to allocate objectives and resources to processes. During this phase people are
trained and empowered to plan and control the provision of services. With progress of
Excellence 2008, Air China is endeavoring to integrate quality into business, and
consistently integrate strategy business functionalities with TQM discipline, aiming to SWOT analysis
transform the company into customer service organisation.
for Air China
References
Puffer, S. and McCarthy, M. (1996), “A framework for leadership in TQM context”, Journal of
Quality Management.
Weihrich, H. (1982), “The TOWS matrix: a tool for situational analysis”, Journal of Long Range 173
Planning, Vol. 15 No. 2.

Further reading
Raynor, M.S. (1992), “Quality as a strategic weapon”, Journal of Business Strategy, Vol. 13 No. 5,
pp. 5-9.
Scholtes, P. and Hacquebord, H. (1988), “Beginning the quality transformation”, Quality Progress,
pp. 77-83.
Schonberger, R.J. (1992), “Is strategy strategic? Impact of total quality management on strategy”,
Academic of Management Executive, Vol. 6 No. 3, pp. 80-7.
Spitzer, R.D. (1993), “TQM: the only source of sustainable competitive advantage”, Quality
Progress, Vol. 26 No. 6, pp. 59-64.
Tata, P. and Prasad, S. (1998), “Cultural and structural constraints on total quality
implementations”, Total Quality Management, Vol. 9 No. 8, pp. 703-8.

Corresponding author
A.M. Ahmed can be contacted at: a.m.ahmed1@bradford.ac.uk

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

BIJ
13,1/2 Benchmarking web site functions
Hugues Boisvert
CMA International Centre, Montreal, Canada, and
Marie-Andrée Caron
174 Department of Accounting, UQAM, Montreal, Canada

Abstract
Purpose – To measure, classify and compare web site functions’ development.
Design/methodology/approach – The objectives were achieved by developing a methodology
to measure, classify and compare web site functions development. The measurement was based
on the presence (or absence) of 91 web site components. The classification was achieved using an
applied correspondence analysis. The comparison was performed with respect to 4,485 company
web sites from two provinces in Canada. A formal procedure involving 50 assistants was
developed to collect data over 5,000 sites within a month period and a conceptual model was
developed to interpret results.
Findings – Findings show that web site functions development could be described on a three
dimensional space, the three axes corresponding to directions of development. The development
status of the whole sample appeared as a cone in which five classes (or categories) of web sites
could be identified and described with respect to their development profile. The development of
18 web site functions was also benchmarked with respect to observations within a class and with
respect to some other characteristics like the industrial sector and the province where the
company was located.
Research limitations/implications – Web site functions development was evaluated only with
respect to components accessible to the general public. Development of intranet and extranet were not
evaluated and hence taken into account for overall web sites development.
Practical implications – This research project of web site functions development was part of a
more comprehensive project aimed at evaluating and documenting the impact of using a web site on
business processes. So, combining the results of the two projects, allowed the authors to advance
suggestions of how web sites should be developed to generate value for companies.
Originality/value – The approach to web site functions development evaluation is original. The
methodology could be replicated anytime. The typology that emerged from the analysis is unique.
Moreover, given the large sample of 4,485 web sites, results are statistically valid.
Keywords Benchmarking, Internet, Worldwide web
Paper type Research paper

Benchmarking: An International
Journal
Vol. 13 No. 1/2, 2006
pp. 174-189
q Emerald Group Publishing Limited
1463-5771
DOI 10.1108/14635770610644664
Rationale
The benchmarking of web site functions was planned as a first phase of a more
comprehensive project[1] aimed at documenting the impact of internet technology on
business processes. This benchmarking exercise was necessary to identify the most
highly developed web sites with respect to specific functions. Companies with such
highly developed web sites were recruited for case studies in the second phase of the
comprehensive project. The rationale for the benchmarking of web site functions is
shown in the conceptual framework (Bégin et al., 2001) in Figure 1.
A web site is a tool that can improve companies’ business process performance,
particularly with regard to activities and tasks with specific objectives: marketing
products and services, selling products, providing after-sales services, etc. Our
 Benchmarking
Environment
Environment

web site
Business

Business context
Business

Industrial sector
Company size functions
Internet strategy

Manage customer accounts 175

Manage external relationships
Manage Human resources
Processes
Business
Processes
Business

Developproducts
Design products
& services

& services

After-sales
Managing

Providing
Market
Produce

Store

Sell

Order
Informational
WebsiteType
WebsiteType

Promotional
Relational Transactional
Promotional-Relational Promotional-Transactional
Figure 1.
Relational-Transactional Conceptual framework

framework identifies 11 such processes that a web site can improve. Moreover,
companies have developed various types of web sites to serve numerous purposes
with respect to business processes. For example, when a company develops a
promotional site, this site will likely be used to market and promote products;
when it develops a transactional site, it will likely be used to sell products, and
when it develops a relational site, it will likely be used to develop preferred ties
with customers, suppliers, employees, investors and numerous other stakeholders.
The rationale behind this benchmarking project is, therefore, to understand web
site development with respect to the functions that a site performs. We also aim to
classify web sites according to their development profile in order to link them to
specific business process improvements that we intend to document in the second
phase of this comprehensive project.

Methodology
First, we posed the following research question:
RQ1. Given that a web site is a tool used to improve the performance of activities
within business processes, how can web sites be measured and compared?
We then established three essential tools:
(1) a method of describing web site functions;
(2) an approach to measure web site development; and
(3) a model to evaluate and compare web site development.

In addition, we needed to plan and implement a data collection and analysis strategy.
BIJ Identification of web site functions
13,1/2 A web site function corresponds to a specific task performed by a web site, with each
task being related to a specific objective. For example, the identification function
corresponds to the task of identifying the company that owns the site, the navigability
function corresponds to the task of making it easier to find information on a site, and so
on. Figure 2 shows the 18 functions benchmarked in this study. Four of the functions –
176 navigability, security, respect and accreditation – were considered support functions
of a site, and the other 14 functions were considered customer value oriented functions
because their objective is either to convey information to customers or directly procure
information from customers. Table I defines the functions of a web site analysed within
this project.
The functions were identified through focus group discussions with experts[2] in
web site design and development. Focus group participants were asked: what are the
usual functions of web sites? Which functions do companies usually develop on web
sites? Which functions of a web site are most useful to companies? The November 2002
project[3] drew significantly on the previous year’s results and analyses[4]. Apart from
the use of an intranet or an extranet, no web site functions were identified by the
participating experts other than those shown in Table I.

Description and measurement of web site functions

The description of web site functions examines how the functions are given concrete
form on a web site? Specifically, how does a web site identify the company to which it
is related? How is a web site made navigable?
Functions are made tangible on a web site through a set of components or elements
of information. For example, the navigability function takes the concrete form of a site
map, a tab bar or navigation menu, a switching functionality, a followed path, online
web site support, an intuitive search engine or any other navigation component.
Table II describes the identified functions in terms of the components of a web site.
Components were also identified by the same panels of experts. Overall, 91 components
were identified by experts. Moreover, the list in Table II was enriched considerably by
the previous year’s findings.

Navigability
Support Security
functions
Respect
Accreditation
Monitoring market
Service to clients

Internal relations

Customer
Client Loyalty
Outmarketing
Identification

Transaction
Expertise

Investors
Products

Services

Contract

Partners
Image

value
Figure 2. functions
Web site functions
 Benchmarking
Objective of the function
web site
Support functions functions
Navigation To facilitate navigation on a site
Security To ensure the security of transactions on a site
Respect To ensure confidentiality and (protect visitors’
interests) 177
Accreditation To confirm the qualifications of a web site
Customer value functions
Identification To identify the company
Image To promote a company’s image of its proper
environmental, social and ethical conduct
Expertise To promote company expertise and brands
Products To market products
Services To market services
External marketing To promote another company’s products or services
Contract To inform clients of transaction-related conditions
Transaction To carry out a transaction: make a sale or receive a
payment
Customer service To support customers with specific services
Client loyalty To develop customer loyalty
Investors To interest and support investors
Partners To develop a personalized relationship with partners
Internal relations To develop preferred relationships with candidates
and employees Table I.
Monitoring and leadership To monitor and lead changes in a community of Definition of web site
customers functions

A component is related to a function if it contributes to that function. For example, a

tab bar clearly contributes to navigability. However, the contribution of any given
component to a function is not always clear cut. Some components may contribute to
more than one function, and components contributing to a function may not all carry
the same weight. For example, Table II shows that the company logo contributes not
only to the identification function, but also to the image function. The reason the
company logo was assigned to the function identification is that the experts on the
panels felt that it contributes more to identifying the company owning the site than to
promoting the company’s image, but it could have been assigned to the image function
with a different weight.
The identification of functions and their definition in terms of components are
useful to interpret results. However, in order to avoid biased results because of
inappropriate assignment of components to functions, the statistical analysis carried
out was based solely on the presence or absence of components, and not on their
assignment to functions. Moreover, the measurement of components is simple since a
component[5] is either present (value 1) or absent (value 0) on a site. The statistical
analysis was, therefore, conducted using a matrix where each line represents an
observation and each column represents a component. Hence, the assignment of
components to functions has no impact on the statistical analysis, although it
facilitates interpretation of the various profiles of web site development.
BIJ
Functions Components
13,1/2
Navigability Site map, tab bar or navigation menu, switching
functionality, followed path, online web site support,
intuitive search engine, other navigational
components
178 Security Entrance in a secure zone, seal certifying security,
payment security
Respect Policy on disclosure of personal information, opt-out
from a mailing list, opt-in to a mailing list, cookies
present
Accreditation Seal certifying the nature of content, seal certifying
customer satisfaction, seal certifying conformity
Identification Company logo, civic address, telephone number,
what we do, history of the company, electronic
address, location of facilities, divisions and
subsidiaries
Image President’s message, mission, values, press review,
press releases, membership in associations, business
partners, link to business partners, social, ecological
and environmental causes, sponsorships
Expertise Portfolio of achievements, clients’ messages, awards
and distinctions, ISO or other accreditation, brands
Products Electronic catalogue, images of company products,
product availability, information on product safety,
instructions on product care, information on product
use, new product announcements, product
promotions, free samples, points of sale
Services Services offered, points of service
External marketing To advertise products or services from companies
with no ties to the company hosting the site
Contract Prices of products available online, currency used for
billing, link to currency rates, choice of currency,
information on shipping costs, information on taxes
and customs, delivery conditions, conditions related
to payment policy, conditions related to guarantees,
cancellation of a sale and exchange policy,
confirmation before payment, conditions related to
financing
Transaction Online purchasing option, online payment option
Customer service Online quote, generic information including technical
and professional advice, technical support, customer
service, transaction history, transaction follow-up,
classified ads
Customer loyalty Personalized client profile, customer survey, contests
and drawings, games and other entertainment
Investors Financial reports, financial indicators, financial
analyses by third parties, share transactions
Partners Access to an extranet, distributor recruitment
Internal relations Section on careers, skills profiles, conditions related
to jobs offered, resume posting, presentation of
Table II. employees, access to an employee intranet
Description of web site Monitoring and leadership FAQ forum, discussion groups regarding products
functions and services
Data collection method Benchmarking
The data collection method involved five main activities: web site
(1) development of a data collection sheet; functions
(2) construction of lists of web site addresses;
(3) establishment of a data collection procedure;
(4) design and implementation of an electronic collection and control system; and 179
(5) recruitment and training data collectors.

A data collection sheet

The data collection sheet is directly derived from the list of components described in
Table II. However, to ensure that each component is interpreted similarly by the
numerous assistants that collected data, we provided a clear and comprehensive
definition of each component. In some cases, we complemented the definition with an
example of what the component looked like on sites where it appeared. After
describing the components, we verified the consistency of the interpretations, and we
re-organized the order in which the components appeared on the sheet to simplify the
data collection process.
In addition to the 91 components, there were five other fields of information: one
identifying the company with a code to ensure privacy, one representing the industrial
sector, on related to company size, one related to the location of the head office, and one
related to the languages of the site.

Lists of web site addresses

We compiled lists of web site addresses of companies from specific industrial sectors
whose head office was situated in the Canadian provinces of Québec and Ontario. We
confirmed the validity of all addresses before the data collection began. Specifically, we
ensured that two addresses did not lead to the same site, that the sites were not under
construction, that the sites did not belong to government agencies or non-profit
organizations, that there were no pornographic sites, and that the sites were all from the
industrial sectors retained. Moreover, we gathered data on company size in terms of the
number of employees, and we ensured that the sample reflected the overall industrial
picture of the Canadian economy in the two provinces.
Table III shows the number of sites evaluated by industrial sector and by province
and Table IV presents the sample population by province and by company size.

Data collection procedure

To enable a large team to collect a sizeable quantity of data, a rigorous data collection
procedure was. required. Moreover, to ensure proper comparison of web sites, we
needed to collect the data within a relatively short period of time given the pace of web
site change. Our objective was thus to collect all the data within a period of four weeks.
Data was collected during November 2002, the same period as the 2001 project.
Preliminary tests revealed that it took approximately 20 minutes to collect data from a
single site. We consequently hired 50 students, divided into teams of six to eight people
working under a supervisor. Supervisors were in charge of answering questions and
validating the data collected by the students on their team. Supervisors reviewed all
the data collected by each student for the first ten web sites. Following that, if they
BIJ
Industrial sector Québec Ontario Total
13,1/2
Agriculture, forestry, fishing and hunting 52 32 84
Construction 142 87 229
Manufacturing 949 796 1,745
Wholesale trade 297 356 653
180 Retail trade 548 365 913
Transportation and storage 77 42 119
Information and cultural industry 129 142 271
Table III. Finance and insurance 92 93 185
Web sites surveyed in Real estate and rental services 71 48 119
November 2002 by Professional, scientific and technical services 45 44 89
province and industrial Other sectors 55 23 78
sector in Canada Total web sites surveyed 2,457 2,028 4,485

Company size Québec Ontario Total

Less than five employees 633 29.5% 261 14.2% 894 22.5%
Between five and nine employees 404 18.8% 317 17.3% 721 18.1%
Between 10 and 29 employees 573 26.7% 552 30.1% 1,125 28.3%
Between 30 and 99 employees 327 15.2% 417 22.7% 744 18.7%
Between 100 and 499 employees 178 8.3% 232 12.6% 410 10.3%
More than 500 employees 33 1.5% 55 3.0% 88 2.2%
Total when size is known 2,148 100% 1,834 100% 3,982 100%
No information on size 309 194 503
Table IV. Total 2,457 2,028 4,485
Web sites surveyed in Less than 500 employees 98.5% 97.0% 97.8%
November 2002 by Less than 100 employees 90.2% 84.4% 87.5%
company size in Canada Less than 30 employees 75.0% 61.6% 68.8%

were satisfied with the students’ interpretation of the components, they would review
the data collected from one out of every three sites for the remaining sites analysed.

An electronic data collection and control system

The entire data collection and control system was automated. Initially, students
received an electronic list of ten addresses from which to collect data. They were also
provided with an electronic data collection sheet. When they collected the data from
their ten assigned sites, the data collected were sent electronically to the supervisor for
validation. Upon validation, another list of ten sites was sent to the student, and so on.
The data validated by supervisors were transferred to a secure database. Following
four weeks of data collection, we had compiled an electronic database of 4,487 lines and
96 columns.

Recruitment and training of students

Students from HEC Montréal[6] were invited to evaluate web sites. While they were
asked to commit to evaluate a minimum of 100 web sites, some students evaluated a
few hundred sites. They were provided with training and were tested with a sample of Benchmarking
web sites before they began collecting data. They were also assigned to a supervisor web site
who met with them, answered questions and provided support as needed during the
data collection period. functions

Main results
The main results are: 181
.
a method to identify dimensions of web site development;
.
a typology of web sites with respect to their development profile; and
.
benchmarks related to function development.

A method to identify dimensions of web site development

The method to identify dimensions of web site development was refined from year to
year as the projects were carried out[7]. The method consists of:
.
A description of web site development in terms of the presence (or absence) of a
set of components (91 components), the list of components being determined a
priori by panels of experts.
.
The use of applied correspondence analysis (ACA)[8] to identify dimensions
(axes) of development, expressed in terms of subsets of components that are
present and subsets of components that are absent.

Application of ACA[9] to the matrix composed of the 4,487 data observations (lines),
describing 91 binary components (columns), enabled us to identify three main
discriminating axes, explaining 24.74 per cent of the total inertia[10] of the
observations. This result is highly meaningful given the fact that all data were of a
binary type. The first axis, dimension 1, accounts for 13.04 per cent of the total
inertia[11], the second axis for 8.41 per cent, and the third axis for 4.02 per cent. Figure 3
shows the observations with respect to the three dimensions defined by the three
discriminating axes.
The co-ordinates of the three axes suggest that the development of the web sites
examined evolves according to three dimensions. Figure 4 shows the plane formed by
axes 1 and 2. Figure 5 shows the plane formed by axes 2 and 3.
Dimension 1, explaining 13.44 per cent of the total inertia of the observations, is
labelled the development axis because one extremity of the axis contains sites with
very few components and at the other extremity of the axis we find sites with a greater
number of components. Therefore, along dimension 1, development refers to the
number of components that are present on a site. Figure 4 reveals a cone: the tip of the
cone corresponds to very poorly developed sites (very few components are present),
and the sites closest to the left-hand side of Figure 4 are more developed (more
components are present). Moving toward the left-hand side of the figure, the
observations appear to be increasingly dispersed. In effect, they are spread out in
relation to axis 2 because the more components there are in a subset, the more likely
they are to differ from one another along axis 2.
Dimension 2 is labelled the relational – transactional axis. A site is transactional
when the transaction function is developed, and a site is considered relational when it
has interactive components other than transactional components. The bottom of
BIJ
13,1/2

182

Figure 3.
The sample projected on a
three dimensional spaces
defined by the three
discriminating axes

Dim 2 –7.60 % Transactional sites

–1.0

Highly developped sites

–0.5

Little developped sites

0.5

Figure 4. 1.0
Plane formed by
2.0 1.5 1.0 0.5 0 –0.5
axes 1 and 2 Dim1 –13.44 %
Relational sites

Figure 4 contains exclusively relational sites. Conversely, the exclusively transactional

sites are situated at the top of Figure 4. In the middle of the figure, toward the left-hand
side lie the sites that are both relational and transactional, and toward the right-hand
side (underdeveloped sites), there are sites that are neither relational nor transactional.
 Benchmarking
web site
functions

183

Figure 5.
Plane formed by axes 2
and 3

Figure 5 shows a view from the bottom of the cone of observations. Dimension 3 is
labelled the promotional type axis because it distinguishes the type of promotion that
is observable on the site. At the top of Figure 5 are the sites that mainly promote
company products and services and at the bottom of the figure we find the sites that
mainly promote the image and the expertise of a company.
To summarize, our first main result is the development and implementation of a
method to identify web site development profiles. We discovered that companies
developed relational, transactional, promotional sites of type 1 and 2, or simply
informational sites. Promotional type 1 refers to the promotion of products and services
and promotional type 2 refers to the promotion of the company image and expertise.
This is a major finding that will be confirmed by further results and interpretations.
Moreover, in applying this analysis to different samples of data, we clarified other
dimensions of web site development[12].

A typology of web sites

After performing ACA, we applied a method called ascending hierarchical classification
(AHC)[13], which suggested that there were five main classes of sites featuring similar
development profiles. Figure 6 shows the number of sites that were assigned to each
class using AHC analysis[14]. Figure 7 shows the location of the centre of gravity of each
class with respect to the plane formed by dimensions 1 and 2, and Figure 8 shows the
centre of gravity of each class with respect to the plane formed by dimensions 2 and 3.

Description of the classes

Interpretation of class 1 (transactional-relational). Class 1 comprises 254 sites, which
corresponds to 5.7 per cent of the sample surveyed. Labelled transactional-promotional,
this class consists of sites that are fully transactional, but that also featured some
relational characteristics. Companies within this class sell their products through the
BIJ sites and accept online payments. Consumers are well informed of the transaction terms,
13,1/2 including delivery terms as well as security levels related to the transactions. The
well-developed functions within this class are transaction, security, respect, contract and
product along with relational type functions internal relations and partners. The image
and navigability functions are also substantially developed.
Interpretation of class 2 (relational). Class 2, relational, encompasses 398 sites,
184 which corresponds to 8.9 per cent of the sample surveyed. This class is fully relational
and is mainly oriented toward the development of relationships with customers,
employees, partners and investors. The most highly developed functions are internal
relations, investors, image and expertise.
Most of the sites in class 2 belong to companies with more than 500 employees, many
of which operate in the Finance and insurance sector. Moreover, sites within this class
are fully bilingual (English and French) and often provide services in other languages
as well.

Figure 6.
An AHC of the sample
surveyed in November
2002

Figure 7.
Classes’ centres of gravity
with respect to dimensions
1 and 2
 Benchmarking
web site
functions

185

Figure 8.
Classes’ centres of gravity
with respect to dimensions
1 and 3

Interpretation of class 3 (promotional-transactional). Class 3, labelled

promotional-transactional, comprises 335 sites, which corresponds to 7.4 per cent of
the sample surveyed. This class includes a majority of companies from the retail sector.
Sites within this class are mainly oriented toward marketing products and services and
offer an online purchase option. The most highly developed functions are product,
contract and transaction, although online payment is not always possible.
Class 4 (promotional-relational). Class 4 comprises 1,821 sites, which corresponds to
40.6 per cent of the sample surveyed. Sites within this class are developed in a
promotional way aimed at marketing products, services or expertise, and also in a
relational way aimed at developing preferred relations with customers in order to
increase their customer base and develop customer loyalty.
The most highly developed functions are products, services, expertise and customer
service. Navigability is also well developed.
Many SME (fewer than 100 employees) are found within this class. They mostly
operate in the manufacturing, construction and wholesale trade sectors.
Class 5 (informational). Class 5 comprises 1,679 sites, which corresponds to 37.4 per
cent of the sample surveyed. This class was labelled informational because these sites
are strictly informational. Only the identification function is well-developed. Often,
sites within this class resemble a virtual business card. Many are run by businesses
with fewer than five employees, and many are French only when the company is
located in Québec and English only for companies located in Ontario.
To summarize, the ACH analysis enabled us to define a typology of web sites, to
define web sites classes according to their development profiles and to classify web
sites within the five classes.
BIJ Benchmarks related to function development
13,1/2 We have already described how to measure web site development in terms of the
presence (or absence) of web site components. However, in order to analyse the
development of functions, it is important to define a description of the functions in terms
of the components already measured. Therefore, given the description of functions in
terms of components, as presented in Table II, we propose an approach to benchmarking
186 function development. The identification function illustrates the approach.
The following three hypotheses are offered:
H1. A web site function can be defined in terms of components of a site.
H2. Various components related to a function may contribute to the function in
different ways.
H3. Not every component related to a function needs to be present for that
function to be active.
Given these hypotheses the development (Di) of a given web site function i can be
written as follows:

Di ¼ Scij pij
where cij is a binary variable (which takes the value of 0 when component j is present
on a site and 0 if not), related to component j of the function i, and where pij is the
weight related to the contribution of component j to function i.
The weights pij are equal to: 1, if component j contributes very little to function i; 2, if
component j contributes slightly to function i; 3, if component j has contributes
moderately to function i; 4, if component j contributes substantially to function i; 5, if
component j’s contribution is essential to function i.
Accordingly, based on the fact that the range of a statistical distribution < 6s, we
used the following scale, described in Table V, to interpret the score of a function.
Figure 9 shows the interpretation of a function’s development.
The development of a site in terms of its functions could be evaluated in a similar
way by summing up each function’s development score.
Development ðsiteÞ ¼ SDi over all function i.
Figure 10 shows the development of the function identification for a sample of 4,487
observations. In this case, the identification function is developed at least at a normal
level for 92.3 per cent of the sites surveyed and only 1.4 per cent of the sites surveyed
have a less than rudimentary development of the identification function.

Function development Score X Interval range

Very rudimentary X , MAX/6 MAX/6

Rudimentary MAX/6 , X , MAX/3 MAX/6
Normal MAX/3 , X , MAX/3 MAX/3
Substantial MAX/3 , X , MAX/6 MAX/6
Table V. Very substantial MAX/6 ,X MAX/6
Interpretation of the
value Di describing the Note: MAX is the highest possible score of a function. Such a score occurs when all components
development of function i describing a function are present on a site
All functions were evaluated with this methodology. In addition, companies can visit Benchmarking
http://web.hec.ca/cicma/en/services/bulletins/dev_web/bull_auto.cfm to benchmark web site
their company web site.
functions
Conclusion
We have presented a methodology to benchmark web site development with respect to
functions or tasks that a web site can perform. The methodology was refined over three 187
projects that spanned a three-year period. Using this methodology, we have defined a
typology of web site profiles and we have established benchmarks of function
development. We have illustrated the methodology with an analysis of 4,487 web sites
from companies located in the provinces of Québec and Ontario, mainly SME from
selected industrial sectors.
This study is based on the premise that a web site is a tool to improve business
process performance. Thus, given the fact that general managers consider a web site as
a tool that can improve some tasks and activities, it is possible to infer, from the results
obtained, the way managers could plan web site development. Firstly, managers must
decide how they would like to use the web site, that is its general purpose. For example,
will the web site be intended to support the development of products and services,
provide after-sales service, manage customer accounts or manage external
relationships. Second, given a specific objective of the web site, managers should
envision development of functions that would contribute to the targeted objective. For
example, if the web site is intended as a promotional tool in the construction sector, the
manager should concentrate on the development of the image and expertise functions.
Third, based on the most developed sites within a class, managers would have to
decide on the subset of components to be implemented within the required functions.
For example, managers planning a relational web site could examine how web sites
within this class have been developed by companies of a similar size and within related
industrial sectors. One limitation of this study is that it does not provide insight into
how components can best be organized on a site.
Normal
Very Very
Rudimentary Substantial
rudimentary substantial

Figure 9.
Minimum –2σ –1σ Median 1σ 2σ Maximum
Interpretation of the
development of a function

Very substantial 14,7%

Substantial 30,2%

Normal 47,5%

Rudimentary 6,3%
Figure 10.
Very rudimentary 0,9% Development of the
identification function
Absent 0,5% using the November 2002
sample
0.0% 10.0% 20.0% 30.0% 40.0% 50.0%
BIJ Notes
13,1/2 1. This paper describes the methodology and results of a third consecutive web site function
benchmarking project over a three-year period. Data for this last project were collected in
November 2002 and the methodology and results presented originate from the November
2002 project. Previous years’ results were published in CMA Management (Boisvert, 2001,
2002a, b, c, 2003a, b, c; Bégin and Boisvert, 2002a, b, c, d, e; Tchokogué and Boisvert, 2002)
188 available at: www.cma-canada.org.
2. Overall, approximately 20 experts involved in web site development participated in the
panels.
3. In the November 2001 edition, we considered only 67 components defining nine functions.
4. Results of the November 2001 edition were published in CMA Management, Special issue on
Canadian web site development, The Society of Management Accountants of Canada,
November 2002 issue, p. 18-32.
5. No attemps were made to evaluate the quality of any component in terms of its
programming on a site.
6. HEC Montréal is the business school affiliated with the University of Montréal (www.hec.ca).
7. In the November 2000 edition, we used the ACA with a sample of 2,725 sites described by a
set of 60 binary components. In the November 2001 edition, we used the same ethod with a
sample of 5,935 sites described by a set of 67 binary components, and in November 2002
edition, we used ACA with a sample of 4,487 sites described by a set of 91 binary
components.
8. ACA determines axes called discriminating axes that minimize the distance between the
observations and the axes according to a x 2 metrics (Lebart et al., 1997).
9. A software SPAD (www.decisia.fr) was used to perform the analysis and to generate the
figures reproduced in this paper.
10. SPAD software (www.decisia.fr) was used to perform the analysis and to generate the
figures reproduced in this paper.
11. The term inertia is assimilated to the one of variance (Clausen, 1998).
12. In effect, we applied the ACA to other data samples, a sample of sites from France and
another one from Brazil that we do not discus in this paper.
13. AHC is so named because it separates any sample of observations into two classes in order to
minimize the inertia (variance) of each class. According to Figure 7, classes 1 and 3 were
initially separated from the three remaining classes. Then, class 1 was separated from class
3, and classes 2 and 4 were separated from class 5. Lastly, class 2 was separated from class 4.
14. SPAD, the same software referenced above was used to perform the ACH analysis.

References
Bégin, L. and Boisvert, H. (2002a), “Making e-commerce profitable”, CMA Management, Vol. 75
No. 9, pp. 28-31.
Bégin, L. and Boisvert, H. (2002b), “Strategically deploying”, CMA Management, Vol. 75 No. 9,
pp. 20-3.
Bégin, L. and Boisvert, H. (2002c), “Deployment of e-commerce: meeting the needs of the
cyberconsumer”, CMA Management, Vol. 76 No. 2, pp. 26-9.
Bégin, L. and Boisvert, H. (2002d), “The internal factors that can make or break e-commerce
implementation”, CMA Management, Vol. 76 No. 2, pp. 22-5.
Bégin, L. and Boisvert, H. (2002e), “E-commerce: evaluating the external business environment”, Benchmarking
CMA Management, Vol. 76 No. 2, pp. 16-21.
Bégin, L., Tchokogué, A. and Boisvert, H. (2001), Strategic Deployment of E-Commerce, IQ editor,
web site
The Communicators Collection, Québec, p. 25. functions
Boisvert, H. (2001), “Reinventing the @nterprise”, CMA Management, Vol. 75 No. 2, pp. 30-3.
Boisvert, H. (2002a), “Smart web development”, CMA Management, Vol. 76 No. 8, pp. 18-22.
Boisvert, H. (2002b), “Provincial divides”, CMA Management, Vol. 76 No. 8, pp. 24-7. 189
Boisvert, H. (2002c), “Buyer beware”, CMA Management, Vol. 76 No. 8, pp. 24-7.
Boisvert, H. (2003a), “Integrating the internet”, CMA Management, Vol. 77 No. 3, pp. 20-2.
Boisvert, H. (2003b), “Starting out: communicator and advertiser web sites”, CMA Management,
Vol. 77 No. 3, pp. 24-7.
Boisvert, H. (2003c), “The next step: developer and seller sites”, CMA Management, Vol. 77 No. 3,
pp. 28-31.
Clausen, S-E. (1998), Applied Correspondence Analysis: An Introduction, Quantitative
applications in the social sciences, Sage University Paper.
Lebart, L., Morineau, A. and Piron, M. (1997), Statistique Multidimensionnelle, 2nd ed., Dunod,
Paris.
Tchokogué, A. and Boisvert, H. (2002), “Maximizing your web site’s value”, CMA Management,
Vol. 75 No. 9, pp. 24-7.

Further reading
Cornet, A. (2001) Working Paper CICMA 01-05, Mise en oeuvre de la stratégie e-commerce:
impacts sur la structure organisationnelle, les processus d’affaires et les politiques de
gestion des ressources humaines, HEC Montréal, Montreal.

Corresponding author
Hugues Boisvert can be contacted at: hugues.boisvert@hec.ca

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

BIJ
13,1/2 Knowledge-based benchmarking
of production performance
B. Denkena, R. Apitz and C. Liedtke
190 IFW – Institute of Production Engineering and Machine Tools,
University of Hannover, Hannover, Germany

Abstract
Purpose – Recently, benchmarking has become a common approach to optimize production
processes by comparing certain aspects of a company with its competitors. However, one of the
biggest challenges is not only to define suitable benchmarking topics and partners, to gather and
statistically evaluate characteristic data, but to derive concrete measures to interpret the results, i.e. to
overcome the revealed weaknesses. The purpose of this paper is to present an already implemented
and successfully used functional benchmarking methodology for production performance of small and
medium batch size processes, that is currently extended by using a knowledge base for reasoning
strategies to semi-automatically support the interpretation of the extracted statistical data.
Design/methodology/approach – A comprehensive approach is presented to develop a new model
for the evaluation of a small to medium-sized enterprise’s production performance using an existing
European database called BETT Benchmark.
Findings – The knowledge-based concept enables sophisticated interpretation strategies to be used
on an already existing base of real company data. The decisive point of the approach presented is to
map production variables on room for improvement by taking varying parameters into account.
Originality/value – The proposed tool is a valuable tool that takes advantage of a statistically
firmed analysis of a substantial database and combines it with the comprehensive expertise of
experienced specialists in the field of performance assessment.
Keywords Benchmarking, Performance measures, Knowledge management, Continuous improvement
Paper type Research paper

Benchmarking: An International
Journal
Vol. 13 No. 1/2, 2006
pp. 190-199
q Emerald Group Publishing Limited
1463-5771
DOI 10.1108/14635770610644673
Introduction
The ability to adjust today’s production to rapidly changing market conditions, to
fulfill customer needs and to accomplish future requirements has become vital for
modern enterprises in order to ensure competitiveness. One of the most talked about
challenges of organizations is customer satisfaction. As such, Zairi (2000) regards it as
a total concept, which encapsulates not only measurement aspects of customer
satisfaction per se, but as a long-term pursuit of improvement, a culture change that
can yield to competitive outcomes of the highest order. Despite of being among the
most important ones, customer satisfaction is just one of the possible keys to improve
performance.
Especially small companies feel intense pressure to improve their performance to
catch up with global market players and rapid technological advances. In the course of
attaining these challenging goals, performance measurement has become an essential
element of production management (Hvolby and Thorstenson, 2000). Performance
The content of this paper is an outcome of the project “Knowledge-based Benchmarking of
Production Performance for Batch-size Production”, which is funded by Deutsche
Forschungsgemeinschaft (DFG) within the project To 56/169-1.
measurement has already been applied since a long time. While Taylor’s (1911) Knowledge-
approach was based on time measurement, the managers of the 1960s mainly used based
short term financial criteria such as quarterly earnings. During the 1980s and 1990s the
situation changed significantly: self assessment, quality awards, benchmarking, ISO benchmarking
9000, activity-based costing, capability maturity model, balanced scorecard,
workflow-based controlling, etc. were the buzzwords which dominated the
discussions in the field of performance evaluation (Kueng, 1998). In recent years 191
benchmarking has successfully proven to be a powerful tool to provide comprehensive
knowledge of the own production’s strengths and weaknesses, which is doubtless one
of the basic requirements in order to improve performance and to assist strategic
decisions.

The nature of benchmarking

Benchmarking has been defined as the search for industry’s best practices that will
lead to superior performance. Interest in benchmarking has exploded since 1979 when
Xerox first introduced it (Camp, 1989). Today benchmarking as a management-tool is
widely used. It has spread geographically to large parts of the world and proliferated in
a variety of manufacturing and service businesses, including health care, government
and education organizations (Camp, 1995).
By nature definitions of benchmarking typically include the comparison with a
corresponding partner as well as some elements of a continuous improvement process.
When addressed to executives one often finds aspects both of “best practice” and
“improvement of performance”. Andersen (1999) defines benchmarking as a process of
continuously measuring and comparing one’s business processes to comparable
processes in leading organizations to obtain information that will help the organization
to identify and implement improvements. Many regard benchmarking as a method to
compare key figures, often financial ones, for the purpose of ranking the organization
in relation to competitors or the industry average. This might have been the main
application of benchmarking earlier, but today it is a far more multifunctional tool that
is widely applicable. The core of the current interpretation of benchmarking is:
.
Measurement of the own and the benchmarking partners’ performance level,
both for comparison and for registering improvements.
.
Comparison of performance levels, processes, practices, etc.
.
Learning from the benchmarking partners to introduce improvements in your
own organization.
.
Improvement, which is the ultimate objective of any benchmarking study.

During the past decade benchmarking has received significant attention, especially
after its inclusion among Malcolm Baldrige Award criteria. Many companies that have
adopted benchmarking practices attained numerous benefits and succeeded in their
businesses (Underdown and Talluri, 2002).

Essentials for assessing and improving performance

It is extremely important for a performance measurement system to enable to a general
survey fast. Today the basis for such a system is generally formed by performance
indicators. A performance indicator is a policy relevant statistics, number or
BIJ qualitative description that provides strategic information about the condition, health
13,1/2 or functioning of an organization. Operational facts and coherences are summarized
and thus described in a compressed meaningful form that enables to evaluate the
organization’s current situation and to predict its future development. Apart from the
assistance with operational problem solutions, performance indicators also serve as
early warning indicators timely signaling evidence of critical developments. Therefore,
192 performance indicators are not just an extensively used tool in operational practice, but
also an indispensable instrument for controlling (Bititci et al., 2002).
In order to evaluate whether an organization is achieving its goals or not, it is
necessary to have an explicit definition of what those goals are and how the
performance in achieving these objectives can be measured. Quite often anything that a
number can be attached to is called a performance indicator. Such questionable
indicators are neither very meaningful nor of much practical help. Even far from it!
Poorly selected performance indicators will rather cause more confusion than they will
help to increase transparency. When measuring performance, an explicit statement of
objectives is indispensable in order to serve as a guide or a reference point. This
comparative dimension permits a value judgment about the organization or process to
be made. It may be a goal or an objective, an absolute standard, or a past value. A
performance indicator, therefore, permits an assessment, judgment or evaluation to be
made about the performance of an organization.
Owing to their nature performance indicators always bear the risk of their
misinterpretation when considered individually. In fact, it is a common mistake
assessing performance developments just by single indicators without taking into
account their interdependencies with other indicators. Thus creating a system of
interdependent and complementary performance indicators will lead to a
comprehensive system that accomplishes the task of providing compressed
information while identifying critical developments by looking at the whole picture
(Maleyeff, 2003).
It is by identifying a range of benchmark values that an indicator develops meaning
and substance. Establishing average and best practice performance based on real data
permits a range to be identified. In doing so, it makes people aware of improvements
that are orders of magnitude beyond what they would have thought possible.
Benchmarking using pre-established indicators as a framework can result in a
continually updated range of performance values. This provides the foundation for a
much more dynamic assessment methodology than does a static set of criteria or rating
protocol. As such, benchmarking contributes to the continual improvement in
performance values, and at a broader scale, to the on-going progress towards a more
sustainable future.
Targets involve the identification of a preferred future performance. One of the most
difficult parts of setting performance targets, however, is justifying that the target
achieves the right blend of feasibility and challenge. By basing targets in part on a
range of existing performance values, it is possible to identify what might be
achievable for any given organization. It is through this process that desired future
performance is identified and can then be translated into policies or actions.
To establish an effective performance measurement system a comprehensive
framework of interdependent and complementary performance indicators is needed to
serve as a solid basis. As a result of the indicator’s constitution, factors such as
corporate image, employee educational background, technical know-how, or quality of Knowledge-
management cannot be regarded in such a performance indicator system. based
Nevertheless, they should also be taken into consideration as explanatory factors
(Andersen and Jordan, 1998). benchmarking
Since operations are in general managed from a functional perspective, modeling
processes is a fundamental step to understanding the flow of information and
resources through the processes of the internal value chain. Modeling can also bring 193
the benefit of helping in assessing performance of operational and supporting
processes (Carpinetti and de Melo, 2002).
Performance indicators are not absolute facts carved in stone. They are best
interpreted as indicative, suggestive or diagnostic. They are essential to the
benchmarking process, however, they themselves do not provide complete answers.
Performance indicators should be used to identify differences between benchmarking
partners, and changes in organizations over time. They act as pointers to identify areas
or subjects for further analysis. It is through this additional focused work that the real
value from benchmarking can be achieved.

Benchmarking small and medium-sized enterprises

Based on the fact that small and medium-sized enterprises (SMEs) constitute at least 95
percent of the enterprises in the European Community, Cassell et al. (2001) focus their
research on the use of benchmarking in SMEs. In spite of the significance of SMEs,
much management theory continues to emphasize large firms. This implicitly assumes
that SMEs are less important than large firms and should rather learn from the
know-how of large firms. However, given the economic significance of SMEs it seems
that investigation of their working practices offers interesting insights to researchers.
Research in this area has demonstrated that SMEs can be rich sources of innovation in
relation to new management methods.
Smaller companies are leaner and show a more decentralized production structure.
Although this situation is beneficial, difficulties exist in realizing structural
improvements leading to lower production costs. SMEs are able to introduce new
products faster to the market (Tönshoff, 2000).
Most SMEs recognize the necessity of new impulses from outside of their own
organization. An important customer might demand better performance, sales might
drop significantly over a short period of time, or government regulations may require
major changes to maintain compliance. Once the need to transform is recognized, the
desire to improve must be established among top management. The top management
must realize that radical change is required to retain the business of an important
customer, to comply with government regulations or to become competitive. Once
managers realize that radical change is a necessity, they look for effective means to
accomplish this. Since many SMEs cannot afford private assistance, such as
consultants, or to hire highly trained professionals, such as a full-time engineer, they
often look to low cost public sources of help (Underdown and Talluri, 2002).

The BETTI method

The BETTIw Benchmark (Benchmark Tool to Improve the Production Performance) is
an exemplary tool that fulfills the needs for a progressive performance measurement
system. It is a method that has been specifically developed by IFW (Institute of
BIJ Production Engineering and Machine Tools, University of Hannover, Germany) and
13,1/2 European partners in order to support SMEs to improve their production performance
(Denkena et al., 2003).
The BETTIw Benchmark is addressed to a specific group of companies. It has been
developed for SMEs producing relatively small volumes in batches. SMEs are defined
as companies that employ 50 to 500 people. Target companies produce products in
194 batch sizes that typically vary between one and several thousands. Main industrial
sectors are mechanical engineering, electrical engineering, and machine building
industry, but companies that, for instance, produce furniture or textile products, are
also within the scope of the benchmark.
Spendolini (1992) introduced a general approach to benchmarking. Using a formal
description he divided the benchmarking process into five phases:
(1) determine what to benchmark;
(2) form a benchmarking team;
(3) identify benchmarking partners;
(4) collect and analyze benchmarking information; and
(5) take action.

Since the BETTIw Benchmark is addressed to certain companies operating in a specific

field, a uniform questionnaire is used to collect the data needed. Thus the object of the
benchmarking analysis is mostly determined. Owing to the fact that employees with
access to production and personnel data are able to fill out the questionnaire, it is easier
to form a benchmarking team.
The time consuming search for adequate benchmarking partners becomes needless
since the BETTIw Benchmark database contains production data and valuations of
more than 120 European enterprises located in 11 countries. The access to
benchmarking could thus be eased for SMEs.
Commonly, there is a distinction between four different types of benchmarking
based on the type of partner (Camp, 1989):
(1) Internal benchmarking. By comparison of performance of units or departments
within one organization. Even though not explicit in this definition, comparison
can also be made of similar products or services of similar business units.
(2) Competitive benchmarking. By comparison of performance with a direct product
competitor. In this case, comparison of products or services and business
processes can be made. Reverse engineering is a term more appropriate for
product benchmarking.
(3) Functional benchmarking. Specific function comparison with best practice. It is
an application of process benchmarking that compares a particular business
function in two or more organizations in the same industry.
(4) Generic benchmarking. Search for the best practice irrespective of industry. It is
similar to functional benchmarking but the aim is to compare with the best in
class without regarding to industry.

The BETTIw Benchmark method not only evaluates SME performance against their
competitors, but also against comparable best performers from other sectors.
Therefore, it focuses on functional and generic benchmarking, although – to some Knowledge-
certain extend – internal as well as competitive benchmarking analysis are possible. based
The BETTIw Benchmark questionnaire collects the data required for the analysis in
a structured way, providing definitions and details on how to present the data. Based benchmarking
on the data collected, a set of performance indicators and explanatory factors is
calculated. These explanatory factors are used to select similar, comparable companies
from the database. In this way, for each performance indicator analyzed three to seven 195
selections are made to evaluate the performance against a group of companies similar
in a certain relevant aspect. Thereby the analysis made provides a broad and complete
overview of the company’s performance including insight of the factors that influence
this performance the most.
Each company receives a detailed feedback report dedicated to its situation and
based on best practice examples that fit into their situation (Teunis and Rietz, 2001).
This report includes indications for improvements and more in-depth analysis based
on the production’s organization and the performance profile.

Knowledge-based benchmarking
As a result of the experiences attained with the BETTIw Benchmark so far, it is the
objective of the approach presented here to develop and implement a new
knowledge-based procedure to evaluate production performance of SMEs. Based on
a comprehensive database with production-data and valuations of more than 120
European companies, this procedure enables to evaluate production performance
within the conditions of a flexible batch size production.
Most existing approaches provide simple profiles of strengths and weaknesses in
the form of quantitative data for further interpretation. Expertise acquired in previous
investigations is neglected or insufficiently used. In contrast, the presented approach
comprises the methodological integration of both quantitative performance indicators
and qualitative appraisals as well as the comprehensive use of substantial expertise.
Thereby the determination of concrete room for improvement, like for instance lead
time reduction or restructuring measures, is significantly optimized.
Initially, the characteristics of production in batch sizes need to be determined in
order to identify the critical factors for success, which serve as assessment criteria. The
data of selected companies is used to provide practical basic conditions.
To define the co domain of the performance indicators, the existing data and
characteristic numbers of a certain number of companies are analyzed for general
trends and predictions. Taking these into account, a statistical analysis of the database
is performed to search for interdependencies among critical factors for success,
performance indicators, and explanatory factors. The statistical analysis is done by
taking advantage of analytical methods like correlation, regression, or variance. The
BETTIw Benchmark database provides enough company data to ensure a statistically
firmed predication.
Usually, a company’s performance is evaluated by its performance indicators with
the aid of the comprehensive knowledge of the consultant. We gather this expertise in a
structured way. Therefore, the valuations, critical factors for success, and the
performance indicators of former benchmarks stored in our database, are used in order
to receive specific combinations of these assessment criteria combined with the results
of their interpretation. These combinations of criteria and interpretation will serve as
BIJ parts of the new evaluation model. This model is the basis for the conceptual design of
13,1/2 a prototypical software tool that accesses the comprehensive data of the BETTIw
Benchmark database. As part of the research project the software is implemented and
validated. Benchmarking results are typically interpreted by a human based on his or
her personal expertise. But as a matter of fact, it is not only expertise that influences
interpretation. There is a number of other “soft” factors that may cause a certain bias.
196 Conventional benchmarking is thus in a complex area of conflict between hard
statistical figures and their “soft” interpretation.
If we talk about “knowledge-based interpretation”, we mean a semi-automatic
support of this decisive procedure by applying correlations stored in a knowledge base
on the found statistical data in order to improve the transfer of abstract results into
concrete actions.
Before discussing the knowledge base and its representation in detail, the term
“knowledge” itself should be well-understood. “Knowledge” and “information” are
quite common terms that are in practice and colloquial language often used
synonymously although they have different meanings. Zack (1999) regards
information as data in a certain context and knowledge as an organized
accumulation of information. Knowledge can thus be seen as the top layer of a
hierarchy of symbols, data, and information. It represents information in a
personalized context that can, e.g. be formed by experience.
Knowledge can be internally or externally available, structured, semi-structured or
unstructured, as well as tacit or explicit (Shin et al., 2001). Tacit knowledge is
knowledge that influences behaviors and/or the constitution of mental states but is not
ordinarily accessible to consciousness (Polanyi, 1974), whereas explicit knowledge can
be articulated and transmitted among individuals. According to Nonaka (1991) tacit
and explicit knowledge can be transferred into each other within certain limits. In order
to achieve the targeted mapping of production figures to concrete directives, only
structured and explicit information that is internally available can be used. However,
as a lot of production knowledge is rather tacit than explicit, this knowledge has to be
transformed in order to use it as a structured model.
The question of computationally representing knowledge in a comprehensive and
efficient way is a rather old one. In 1958, Mc Carthy (1958) who also wrote LISP as one
of the most influential programming languages for artificial intelligence applications,
developed a program called “Advice Taker” which used modeled knowledge to search
for solutions to problems. The system used domain knowledge and simple axioms to
generate plans. As one of the first of its kind, it embodied knowledge representation
and reasoning, and manipulated knowledge representation with deduction. However,
expressiveness and representation capabilities were still quite limited. The use of
knowledge for reasoning processes soon called for more powerful knowledge
representation schemes. Within the 1960s and 1970s a number of knowledge
representation languages was developed such as predicate calculus, semantic
networks and frames.
Semantic nets in their original form are unstructured graphs of nodes and links
which offer neither semantics to support interpretation nor axioms to support
reasoning (Woods, 1975). Frames can be seen as structured semantic nets with an
object-oriented description that allow to build, e.g. class-subclass taxonomies (Minsky,
1974). The initial frame concept was soon extended (Bobrow and Winograd, 1977).
Both semantic nets and frame systems are still important today. The common Knowledge-
knowledge base and ontology editor Protégé, e.g. is basically a frame system (Noy et al., based
2000). Current work includes knowledge representation languages such as XML and
RDF (Lassila and Swick, 1999) and their extension DAML/OIL (Fensel et al., 2000) and benchmarking
regards knowledge representation as the key technology for the Semantic web (Denker
et al., 2001).
However, if we regard knowledge-based benchmarking, we deal with a 197
self-contained domain and only a rather restricted number of information classes
and instances. Furthermore, we do not necessarily have to exchange information with
third parties, and our knowledge base is a quite static one that is slowly evolving. On
the other hand, a classical rule-based approach in terms of explicit correlating
combinations of causes and effects will nevertheless soon reach its limits due to the
vast amount of possible combinations of performance indicators, explanatory factors
and possible countermeasures. Thus the representation mechanism can, e.g. ignore
questions of information integration; nevertheless it has to allow probabilistic
reasoning over the implemented knowledge base.
Our new concept takes advantage of integrating both quantitative and qualitative
company information as well as extensive know-how. As a result, one receives
combinations of performance indicators and explanatory factors that in return affect
certain factors of success. Linked with statistically determined interdependencies these
patterns form specific parts of a production structure, each with its own unique
interpretation. The combination of these specific parts builds the core of our new model
for evaluation of production performance.

Conclusion and outlook

It is the objective of the comprehensive approach presented here to develop a new
model for the evaluation of SME’s production performance using an existing European
database called BETTIw Benchmark. Based on current company data this procedure
enables to deduce measures to increase production efficiency.
The decisive point of the approach presented is to map production variables on
room for improvement by taking varying parameters into account. To face this
challenge, a determination and formal description of statistical coherences between
company data, performance indicators, explanatory factors, and critical factors for
success as well as the detailed interpretation of this knowledge is done. A standardized
formulation of such interpretation coherences, their storage and rule-based application
allows fundamentally new insights regarding analyzing and benchmarking production
performance.
The proposed model does not aim at substituting competent benchmarking
consultants. It is a valuable tool that takes advantage of a statistically firmed analysis
of a substantial database and combines it with the comprehensive expertise of
experienced specialists in the field of performance assessment. Users are enabled to
discover complex coherences within company measures and beyond it to receive
detailed recommendations on actions that will improve production performance. Thus
this tool assists the versed expert in his decision making process whereby the
effectiveness of his actual recommendation will be invigorated. Future research will
focus on the task of applying machine learning strategies into the knowledge base in
order to further increase the capabilities of decision support.
BIJ References
13,1/2 Andersen, B. (1999), “Industrial benchmarking for competitive advantage”, Human Systems
Management, Vol. 18 Nos 3/4, pp. 287-96.
Andersen, B. and Jordan, P. (1998), “Setting up a performance benchmarking network”,
Production Planning and Control, Vol. 9 No. 1, pp. 13-19.
Bititci, U., McCallum, N., Bourne, M., MacBryde, J. and Turner, T. (2002), “Performance
198 indicators for sustainable competitive advantage: the next frontier”, paper presented at
Second International IFIP Workshop, Hannover, June, pp. 2-11.
Bobrow, D.G. and Winograd, T. (1977), “An overview of KRL, a knowledge representation
language”, Cognitive Science, Vol. 1 No. 1, pp. 3-46.
Camp, R.C. (1989), Benchmarking: The Search for the Industry Best Practice that Leads to
Superior Performance, ASQC Quality Press, Milwaukee, WI.
Camp, R.C. (1995), Business Process Benchmarking: Finding and Implementing Best Practices,
ASQC Quality Press, Milwaukee, WI.
Carpinetti, L.C.R. and de Melo, A.M. (2002), “What to benchmark? A systematic approach and
cases”, Benchmarking: An International Journal, Vol. 9 No. 3, pp. 244-55.
Cassell, C., Nadin, S. and Older Gray, M. (2001), “The use and effectiveness of benchmarking in
SMEs”, Benchmarking: An International Journal, Vol. 8 No. 3, pp. 212-22.
Denkena, B., Apitz, R. and Liedtke, C. (2003), “Knowledge-based benchmarking of production
performance”, in Putnik, G. and Gunasekaran, A. (Eds) Business Excellence ’03,
Guimaraes, May, pp. 166-71.
Denker, G., Hobbs, J.R., Martin, D., Narayanan, S. and Waldinger, R. (2001), “Accessing
information and services on the DAML-enabled web”, paper presented at the Second
International Workshop on the Semantic Web, available at: www.ai.sri.com/ , martin/
papers/semanticweb01.pdf (accessed November 23, 2003).
Fensel, D., Horrocks, I., van Harmelen, F., Decker, S., Erdmann, M. and Klein, M. (2000), “OIL in a
nutshell”, Knowledge Acquisition, Modelling and Management, Springer, New York, NY,
pp. 1-16.
Hvolby, H.H. and Thorstenson, A. (2000), “Performance measurement in small and medium-sized
enterprises”, presented at the Third Conference on Stimulating Manufacturing Excellence
in Small and Medium Enterprises, Coventry, April, pp. 324-33.
Kueng, P. (1998), “Supporting BPR through a process performance measurement system”, paper
presented at BITWorld’98 Conference, New Delhi, February, pp. 422-34.
Lassila, O. and Swick, R.R. (1999), “Resource description framework (RDF) model and syntax
specification”, W3C Recommendation 22 February, available at: www.w3.org/tr/1999/
rec-rdf-syntax-19990222/ (accessed November 15, 2002).
Mc Carthy, J. (1958) in Minsky, M. (Ed.), Programs with Common Sense, MIT Press, Cambridge,
MA, reprinted in Semantic Information Processing.
Maleyeff, J. (2003), “Benchmarking performance indices: pitfalls and solutions”, Benchmarking:
An International Journal, Vol. 10 No. 1, pp. 9-28.
Minsky, M. (1974), A Framework for Representing Knowledge. The Psychology of Computer
Vision, MIT-AI Laboratory Memo 306.
Nonaka, I. (1991), “The knowledge creating company”, Harvard Business Review, pp. 96-104.
Noy, N.F., Fergerson, R.W. and Musen, M.A. (2000), “The knowledge model of Protege-2000.
Combining interoperability and flexibility”, paper presented at the Second International
 Conference on Knowledge Engineering and Knowledge Management (EKAW’2000), Knowledge-
Juan-les-Pins.
Polanyi, M. (1958/1974), Personal Knowledge: Towards a Post-Critical Philosophy, University of
based
Chicago Press, Chicago, IL. benchmarking
Taylor, F.W. (1911), The Principles of Scientific Management, Dover Publications, New York, NY,
1998.
Teunis, G. and Rietz, W. (2001), “Best practice profile für KMU”, Management und Qualität, 199
No. 12, pp. 42-5.
Tönshoff, H.K. (2000), KMU und Benchmarking: Wettbewerbsfähigkeit steigern durch
internationalen Vergleich, TCW Transfer-Centrum GmbH, München.
Shin, M., Holden, T. and Schmidt, R.A. (2001), “From knowledge theory to management practice:
towards an integrated approach”, Information Processing and Management, No. 37,
pp. 335-55.
Spendolini, M. (1992), The Benchmarking Book, American Benchmarking Association,
New York, NY.
Underdown, R. and Talluri, S. (2002), “Cycle of success: a strategy for becoming agile through
benchmarking”, Benchmarking: An International Journal, Vol. 9 No. 3, pp. 278-92.
Woods, W.A. (1975), “What’s in a link: foundations for semantic nets”, in Bobrow, D. and Collins,
A. (Eds), Studies in Cognitive Science, Academic Press, New York, NY.
Zack, M.H. (1999), “Managing codified knowledge”, Sloan Management Review, Vol. 40 No. 4,
pp. 45-58.
Zairi, M. (2000), “Managing customer satisfaction: a best practice perspective”, The TQM
Magazine, Vol. 12 No. 6, pp. 389-94.

Corresponding author
C. Liedtke can be contacted at: liedtke@ifw.uni-hannover.de

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

BIJ
13,1/2 B2B international internet
marketing
A benchmarking exercise
200
R. Eid
Wolverhampton Business School, Wolverhampton, UK
M. Trueman
University of Bradford School of Management,
Bradford, UK, and
A.M. Ahmed
The European Centre for Total Quality Management,
University of Bradford School of Management,
Bradford, UK

Benchmarking: An International
Journal
Vol. 13 No. 1/2, 2006
pp. 200-213
q Emerald Group Publishing Limited
1463-5771
DOI 10.1108/14635770610644682
Abstract
Purpose – To provide the foundation for further research and good practice in benchmarking since it
combines the best aspects of traditional and new internet marketing activities.
Design/methodology/approach – A methodological triangulation approach has been adopted that
uses secondary data, a mail questionnaire survey, and case studies, to collect quantitative and
qualitative data. The answers of each respondent were coded into the statistical package for social
science (SPSS). Frequency analysis was used for data reduction purposes and to develop an overall
understanding of the survey responses and a general picture of how the sample group has responded.
The initial analysis used descriptive analysis for the whole sample. Furthermore, factor, regression
and path analysis were used to study the research variables and their relationships.
Findings – Identification of business-to-business international internet marketing (B2B IIM) critical
success factors (CSF) by a holistic literature review and a series of statistical and qualitative analysis
of the relevant data. Provides information and valuable insights into the marketers’ acceptance of the
internet for B2B marketing activities. It has also tested the validity of the TAM and IDT in the domain
of internet environment.
Research limitations/implications – B2B IIM is an area of research where theory is still
inadequate. This study is limited by its cross sectional nature. An additional limitation of the study is
the reliance on the subjective, self-report indicators to measure the research constructs in the survey
questionnaire. Another limitation to this study is in representativeness and generlisability issues.
Practical implications – In terms of academic implications, it proposes factors that affect the
adoption of the internet for B2B international marketing activities. A key managerial implication is the
effect of internet usage on B2B IIM efficiency. Notably, it was found that the use of the internet affects
B2B IIM efficiency not directly, but indirectly, through B2B marketing and customer relations
activities, international marketing targeting activities and marketing performance. This means that
the competitive advantage of marketing efficiency is not automatically achieved with the adoption of
the internet.
Originality/value – This paper offers several avenues for investigating individual factors that have
a direct impact on the use of internet for international marketing purpose.
Keywords Business-to-business marketing, Internet marketing, Benchmarking, Critical success factors,
Competitive advantage
Paper type Research paper
Introduction International
The main challenge for many companies today is to manage, control and advance daily internet
business activities and find leverage points to improve performance, particularly in terms
of market-oriented activities that can be influenced by the internet. For example, Avlonitis marketing
and Karayanni (2000) note the central role of the sales force, Chan and Swatman (2000)
observe the involvement of suppliers and customers: and Duggan and Devenery (2000)
stress the importance of integrating “off” and “online” marketing efforts, language and 201
customer culture. But although the last few years have witnessed phenomenal growth in
web-based business activities, many studies show that benefits from internet use is
considerably less than expected (Alshawi et al., 2003). However, one of the most important
developments has been the explosion of international marketing activity and the
associated emergence of the global information superhighway. Consequently marketing
academics and practitioners have devoted considerable effort to identifying the critical
success factors (CSF) for business-to-business international internet marketing (B2B IIM).
Yet few can provide strong theoretical or statistical support in this field, often because they
are of an exploratory or purely theoretical nature.
Some of these issues can be addressed through benchmarking studies, and for the
purpose of this research we define benchmarking as “an ongoing process of measuring
and improving products, services and practices against the best that can be identified
worldwide” (McGaughey, 2002). It includes the application of the skill of comparison –
comparing one’s own performance of a particular strategy, task or operation with that of
others, and provides an objective analysis of how successful is an organisation’s
performance by adopting a systematic measurement process of the improvement (Shen
et al., 2003). It is widely believed that competition provides an impetus for improvements
and benchmarking is a powerful tool for continuous improvement (Zairi, 1998). To this
end a best practice benchmarking approach has been adopted by this research to
examine internet marketing in today’s successful companies (Eid et al., 2002).
The main objectives of this paper are threefold; first to identify CSF for B2B IIM
through an analysis of the literature and secondary case studies; second to empirically
test these factors using a large scale sample representing different industries; and third
to adopt a systematic measurement process from a benchmarking perspective. To this
end, various articles, empirical research, and secondary case studies about B2B IIM
have been reviewed. This material was used to construct the research design and
methodology, develop a survey questionnaire, and interpret the findings. A total of 123
(59 per cent) UK industrial companies replied to a sample frame of 250 taken from the
Yahoo database. As a result 33 factors, drawn from statistical package for social
science (SPSS) factor analysis, showed a direct relationship with successful B2B IIM.
These have been classified into five categories:
(1) Category A: marketing strategy factors
(2) Category B: web site related factors
(3) Category C: global dimension factors
(4) Category D: internal company factors
(5) Category E: external related factors

The constituent variables and interpretation of these findings are discussed below
(Figure 1).
BIJ
13,1/2

202

Figure 1.
B2B CSFs

Category A: marketing strategy related factors

Although could be argued that the internet is merely a resource issue for companies, it
should be an integral part of marketing strategy if it is to reinforce overall corporate
strategy (Chaffy et al., 2000). In fact many researchers consider marketing factors as a
vital component of successful B2B practice (Damanpour, 2001; Chaffy et al., 2000;
Clarke and Flaherty, 2003; Gurau et al., 2001; Hofacker, 2001; Honeycutt et al., 1998;
Lynn et al., 2002; Naude and Holland, 1996; Perry and Bodkin, 2002; Porter, 2001;
Quelch and Klein, 1996; Wilson and Abel, 2002). From a marketing strategy
perspective, this research found that the most important factor to be “top management
support”, followed by “setting strategic goals”, “integrating internet marketing with
marketing strategy”, “collaboration with strategic partners”, and “deciding the
potential audience”. Here it is not surprising to see the significance of “top management
support”, yet if all these factors are combined they represent a balanced formula for
integrating the internet on the one hand, and collaborating with strategic partners and
customers on the other, thereby linking technological processes with strategic goals.

Category B: web site related factors

Companies often use their web sites to convey promotional messages and as a vehicle
to attract interest and inform existing or potential customers. The web site can sell
products directly to business customers, and provide them with a range of electronic or
communication services (Hamill and Gregory, 1997, p. 25). However, since the
corporate web site is a marketing showpiece for the company, many researchers have
considered it as a crucial component of any B2B practice (Anderson, 1996; Herbig and International
Hale, 1997; Hofacker, 2001; Karayanni and Baltas, 2003; Kolokotronis et al., 2002; Lynn internet
et al., 2002; Kotab and Helsen, 2000; Samiee, 1998; Quelch and Klein, 1996). This
research found that a “well-designed site”, that is “effectively marketed”, “easy to find”, marketing
“up to date”, “clear”, and “accurate” are the most important factors. However, attention
is also given to “ease” and “speed” of use. In other words, although some companies
may consider that site appearance is of paramount importance, for new and existing 203
B2B customers, web site design should reflect use as much as if not more so than
appearance, since some sites may appear attractive but may be difficult to use. These
findings are significant because all companies in the sample can be seen as customers
as well as vendors in a B2B context.

Category C: global market related factors

The ability of the internet to make sales to new overseas markets is often used to
champion its use for corporate business. But in practice this depends on many factors
if companies are to succeed in the global market place. For example, Damanpour
(2001) stresses the importance of continuing to meet existing client demands and
fulfil their obligations. However, many authors have considered “global market
related factors” as a vital component of successful B2B IIM efforts, although there are
many pitfalls associated with developing a global presence not least effective
communications in different languages and cultures, and the need for an effective
infrastructure to meet supply and demand requirements (Chaffy et al., 2000; Chan and
Swatman, 2000; Duggan and Devenery, 2000; Gogan, 1997; Hamill and Gregory, 1997;
Herbig and Hale, 1997; Hofacker, 2001; Kotab and Helsen, 2000; Samiee, 1998; White,
1997). This research reinforces previous work in finding the following factors to be
significant – an “in-depth understanding of foreign marketing environment”;
adequate “resources for global operations”; and a “multi-language web site” to reach
target customers. There are also “cultural considerations”, and the “availability of
delivery channels” to overcome the logistics and meet the needs of a complex,
fragmented global market.

Category D: internal company related factors

The importance of delivery channels highlights the need for an appropriate, internal
company infrastructure to manage, control and advance daily activities, and
collaboration with global networks. To this end Bremer (1996) points out the need for
appropriate, planned staff training and awareness. Subsequently researchers have
stressed the importance of internal company environment factors as a critical
component for successful B2B IIM (Avlonitis and Karayanni, 2000; Chan and
Swatman, 2000; Duggan and Devenery, 2000; Gogan, 1997; Huff et al., 1998; Samiee,
1998; Quelch and Klein, 1996). This survey found that there is a need to blend
“technological infrastructure”, and “appropriate internal culture”. The “relationship
between marketing and IS/MIS departments” is also important, together with a “well
trained sales force” if B2B internet marketing is to be effective. This need to focus on
internal processes is reinforced by Alshawi et al. (2003) who observe a strong need to
improve internal organisation and infrastructure if companies are to be responsive and
succeed in using global ICT (information communications technology or information
systems), to gain a competitive advantage.
BIJ Category E: external business related factors
13,1/2 External factors such as “security” can present considerable problems (Cronin, 1996a),
although this research has found that financial security has become less of an issue for
many companies with recent technological progress and closer collaboration between
companies. In fact Furnell and Karweni (1999) observe the need for the development of
strong inter-organisational relationships, and many researchers consider factors
204 related to managing and controlling the external business environment as a key
component of successful B2B IIM (Chan and Swatman, 2000; Duggan and Devenery,
2000; Merrilees and Fry, 2003; Poon and Jevons, 1997; Quelch and Klein, 1996;
Ratnasingham, 1998; Urban et al., 2000; Wilson and Abel, 2002; Zairi, 2001a, b). The
external business environment is a natural extension of the internal company
infrastructure but, as discussed above, will not be accessible without responsive and
well-organised internal processes. This research found that both company and
customers require elements of “trust”, “security” as well as “easy and affordable access
to the internet”. Both will value the building of a “successful relationship” and
“customer acceptance” of policies and processes for the advancement and
sustainability of B2B IIM. This also illustrates the importance of creating a
company culture that is responsive and changes according to new technological
opportunities as well as customer requirements (Alshawi et al., 2003). At the same time
they observe a “universal information language” and companies that can “trade
information” as well as participate in learning.

Research methodology, population and sample

Following an extensive literature review this research was able to divide the key issues
of B2B IIM into the three categories of internet “adoption”, “implementation” and
“implications”. At the same time, because much of the previous work was theoretical,
exploratory or had a very narrow focus, it was decided to carry out a large-scale,
cross-industrial survey of UK companies to empirically record current practice in B2B
internet marketing. Once the results of the survey have been analysed, the results can
be verified by conducting case studies some deliberately contrasting companies.
However, the main focus of this paper is on the major survey results and interpretation.
A research survey questionnaire was designed to incorporate factors relating to the
adoption, implementation and implication variables that arose from the literature. The
survey form was piloted with six companies in January 2002 and points were clarified
by follow-up phone calls after receiving the replies. Revisions were then made to the
questionnaire design and the formal survey was conducted between April and July
2002. A total of 250 B2B companies were randomly selected from the Yahoo database
to form the survey sample and a total of 123 valid returns were received, representing a
59 per cent response rate (Table I). This response is much higher than other similar
surveys conducted in this field, partly because of careful preparations made to the

Total number of questionnaires 250

Completed and returned questionnaires 128
Table I. Unreachable companies 9
Survey response Declined participation 24
summary Response rate 59 per cent
survey questionnaire to ensure clarity and ease of completion, – it took not only 20 International
minutes, – but also by using “follow-up” reminders and phone calls to encourage internet
respondents to complete the forms. A personalised cover letter and a self-addressed
stamped envelope were enclosed with each copy of the questionnaire sent out. Each marketing
organisation was contacted up to five times via e-mails and phone calls before the firm
was dropped from the sample. In fact most companies were likely to perceive their
participation in the study as further evidence of their success regarding using the 205
internet for marketing purposes. Finally care was taken to discover accurate and up to
date respondent names and roles for the “mail out”.
Companies involved in the study remain anonymous, but each has been carefully
logged and recorded for analyses purposes. A total of 50 multiple-choice questions
were asked, which were grouped into three sections: background information (used for
internet adoption), internet usage (implementation), and CSF (implications).

Profile of respondents in the survey

Detailed descriptive statistics relating to the responding organisations and
respondents are shown in Table II.
This shows that more than half of respondents are marketing managers, one
quarter are sales and the remainder are export or IT managers. Only 18 per cent have
more than 500 employees, and 45 per cent have less than 100. This is a considered to be
a reliable set of data since it is a representative profile of UK companies who currently
practice internet marketing, illustrated by a plethora of medium and small companies,
and a few but significant large companies. In terms of internet usage, the minority (18
per cent) have less than two year experience, and nearly a quarter (24 per cent) have
between two and three years, whereas nearly one third (31 per cent) have more than
four years. In other words this reveals the “newness” of the technology and its adoption
for internet use. As far as industry type is concerned, the two largest sectors are
industrial supplies and computing representing two thirds of the sample and possibly

Measure Items Frequency Percent

Position Marketing manager 68 55.3

Sales manager 32 26.0
Export manager 11 8.9
IT manager 12 9.8
Number of employees Less than 100 55 44.7
100-500 46 37.4
501-1,000 16 13.0
More than 1,000 6 4.9
Years of internet use Less than 2 years 22 17.9
2-3 years 29 23.6
3-4 years 34 27.6
More than 4 years 38 30.9
Type of industry Aerospace 8 6.5
Agriculture 14 11.4
Chemical and allied products 8 6.5
Computers 40 32.5 Table II.
Industrial supplies 41 33.3 Characteristics of
Textile 12 9.8 the sample
BIJ those with a “new technology” focus, these are augmented by aerospace, agriculture,
13,1/2 chemical products and textiles which may have a more “traditional” approach.

Instrument validity and reliability

The discriminant validity and data reduction of research findings was assessed using
SPSS principal component factor analysis. The resulting 33 items (variables) defined
206 the five categories of CSF from Category A: marketing strategy to Category E: external
related factors described above. Eigenvalues and scree plots were used to determine
the number of factors to be extracted. Moreover, in order to ensure the valindity of
factor analysis, the Bartlett test of sphericity (BTS) and Kaiser-Meyer-Olkin (KMO) test
of appropriateness were carried out accordingly (Table III). The results (the BTS
ranged from 373.055 to 825.768 and the level of significance at P ¼ 0.000) indicated
that the data are appropriate for the purpose of factor analysis. Statistically, this means
that there are significant relationships between the variables and that they can be
appropriately included in the analysis (Bryman, 1989). As shown the result of sampling
adequacy ranged from 0.767 to 0.907 which, following KMO measure reflecting a high
level of sampling adequacy.
The 33 items or “variables” were loaded onto the five factors using an eigenvalue
greater than 1, and the extracted factors account for a range from 60.272 to 75.667 of
the total variance, using a verimax rotation. All factors loading were higher than 0.5
since, as Hair et al. (1998) observe, a factor loading higher than 0.35 is considered
statistically significant at an a level of 0.05. Only the Category B, web site related
factor of “real time online interactive elements”, had a factor loading less than the
accepted level, so it was ignored in the analysis. The reliability of all constructs was
assessed by the Cronbach a reliability coefficient and exceeded Nunnally (1978)
standards for research (Table IV).

Statistical results of the questionnaire survey

Respondents were asked to assess the CSF distilled from the B2B internet marketing
literature and previous case studies, on a Likert scale of 1 ¼ not critical, 2 ¼ minor,
3 ¼ moderate, 4 ¼ major, 5 ¼ critical. As mentioned earlier, the factors were

KMO Bartlett’s test

Number of Percentage of
extracted Eigen variance
Chi-square df Significance item(s) values explained

Marketing strategy
related factors 0.898 738.520 28 0.000 1 5.272 65.906
Web site related
factors 0.907 825.768 45 0.000 1 6.027 60.272
International
dimension related
factors 0.767 373.055 10 0.000 1 3.374 67.489
Internal related
factors 0.863 443.447 10 0.000 1 3.783 75.667
Table III. External related
KMO and Bartlett’s test factors 0.807 376.726 10 0.000 1 3.454 69.089
categorized into five groups, namely Category A: marketing strategy factors (MSF), International
Category B: web site factors (WSF), Category C: global-related factors (GRF), Category internet
D: internal, company-related factors (IRF) and Category E: external, business-related
factors (ERF). In this way nearly all aspects of B2B IIM have been addressed. marketing

Rank order of critical success factors

A rank order of CSF is shown in Table V. This displays the top 26 most critical factors, 207
ranked on a range of between 3.5 and 4.7, and representing a high level of criticality. Of
these, 14 variables are rated above 4 and can be said to be very critical. In fact the first
seven are all Category B, “web site related” factors such as an easy to find, clear site
that is accurate, up to date and informative. The site should be well designed with a
“speedy” access. The second seven variables are a mixture of Categories A and C,
marketing and external related factors including top management commitment, the
need to integrate internet marketing with marketing strategy and set strategic goals
for internet use. This section also includes easy and affordable access to the internet,
successful relationship with customers and suppliers and one Category B factor that
addresses effective offline marketing of the site. In other words it illustrates a need to
focus on company and customer web site requirements in terms of interaction and
communication, supported by a clear strategy, resourced and embedded within the
company.
The next seven factors are a mixture of categories and reflect efficient processes as
well as stakeholder relationships. They include an in depth understanding of foreign
cultures and markets, supported by the necessary resources to work globally, as well
as security, trust and the effective online marketing of web sites. These are followed by
a group of variables that illustrate some practical considerations such as building
effective relationships between marketing and MIS, the company’s technology
infrastructure, its internal culture and ability to collaborate with strategic partners.
The final section indicates the important role played by the sales force, which
should be regularly trained to keep abreast of new developments, as well as the need to
collaborate with suppliers, distributors and technology providers. Findings regarding
the sales force reinforce previous research that has found that salespeople can be
replaced by technology in some cases (Rich, 2002). However, the sales force should
focus on activities that add value to customer relationships in ways that the internet
cannot.
An one-sample test was conducted to determine whether these observed means of
the CSF (Table V) are significantly different from the mid-point 3.0. The results are
given in Table VI. Here the results are found to be significantly different from the
mid-point 3.0 ( p , 0.01). This confirms that all the critical factors for B2B IIM are
positive, except real time online interactive elements.

Constructs Number of items a

Marketing strategy related factors 8 0.9249

Web site related factors 10 0.9088
International dimension related factors 5 0.8732 Table IV.
Internal related factors 5 0.9187 Measure of constructs’
External related factors 5 0.8782 reliability
BIJ
Factor Group Mean * Rank
13,1/2
Easy to find site WSFb 4.6667 1
Clear site WSF 4.5041 2
Up to date site WSF 4.5041 3
Accurate site WSF 4.4959 4
208 Informative site WSF 4.4715 5
Well designed site WSF 4.4634 6
Speed (download) site WSF 4.3902 7
Top management commitment MSFa 4.3333 8
Customer acceptance ERFe 4.2439 9
Easy and affordable access to the internet ERF 4.1626 10
Integrating internet marketing with marketing
strategy MSF 4.1545 11
Setting the strategic goals for using the internet MSF 4.0813 12
Effective offline marketing of the site WSF 4.0650 13
Successful relationship with customers and suppliers ERF 4.0407 14
In-depth understanding of foreign marketing
environment GRFc 3.9593 15
Resources required to working globally GRF 3.9593 15
Security ERF 3.9512 16
Effective online marketing of the site WSF 3.8537 17
Deciding the potential audience MSF 3.8293 18
Culture considerations ERF 3.7317 19
Trust ERF 3.7236 20
Effective relationships between the marketing and
IS/MIS IRFd 3.6992 21
Collaboration with strategic partners MSF 3.6829 22
Availability of international delivery GRF 3.6667 23
Technology infrastructure IRF 3.6504 24
Internal culture IRF 3.5772 25
Multilanguage web site to successfully reach target
customers GRF 3.5528 26
The great role of the sales force IRF 3.4797 27
The importance of training programs for sales force IRF 3.4390 28
Collaboration with strategic suppliers MSF 3.3496 29
Collaboration with strategic technology provider MSF 3.3415 30
Collaboration with strategic distributors MSF 3.3333 31
Real time online interactive elements WSF 2.1707 32
Table V. Notes: *Mean is based on the five point Likert scale; a(MSR) marketing strategy related factors;
b
Ranking of B2B IIM (WSF) web site related factors; c(GRF) global dimension related factors; d(IRF) Organisation: internal
success factors related factors; e(ERF) Organisation: external related factors

Discussion of findings: strategic factors

Generally, our findings are in line with conventional theory about B2B IIM but add
reliability, rigour and insight to the body of knowledge on this subject. This study
helps us to identify and prioritise CSF in the adoption and implementation of internet
marketing as well as highlight the implications of internet use in a B2B context. At the
same time, companies can develop a benchmarking approach to manage dynamics at
each category level based on the findings and interpretation of this research.
 International
95 per cent internet
Test value ¼ 3 confidence interval marketing
of the difference
Significance
t df (two-tailed) Mean difference Lower Upper
209
Top management commitment 16.906 122 0.000 1.3333 1.1772 1.4895
The strategic goals for using the
internet 12.676 122 0.000 1.0813 0.9124 1.2502
Integrating internet with
marketing strategy 13.015 122 0.000 1.1545 0.9789 1.3301
Collaboration with strategic
partners 6.956 122 0.000 0.6829 0.4886 0.8773
The strategic technology provider 3.156 122 0.002 0.3415 0.1273 0.5556
Deciding the strategic distributors 3.298 122 0.001 0.3333 0.1332 0.5334
Deciding the strategic suppliers 3.451 122 0.001 0.3496 0.1491 0.5501
Deciding the potential audience 9.071 122 0.000 .8293 0.6483 1.0102
Web site related factors 20.341 122 0.000 1.1585 1.0458 1.2713
Well designed site 22.307 122 0.000 1.4634 1.3335 1.5933
Easy to find site 31.015 122 0.000 1.6667 1.5603 1.7730
Up to date site 23.259 122 0.000 1.5041 1.3761 1.6321
Clear site 23.259 122 0.000 1.5041 1.3761 1.6321
Accurate site 21.787 122 0.000 1.4959 1.3600 1.6319
Speed (download) site 18.025 122 0.000 1.3902 1.2376 1.5429
Informative site 21.446 122 0.000 1.4715 1.3357 1.6074
Real time online interactive
elements 29.375 122 0.000 20.8293 21.0044 2 0.6542
Effective online marketing of the
site 7.852 122 0.000 0.8537 0.6384 1.0689
Effective offline marketing of the
site 11.555 122 0.000 1.0650 0.8826 1.2475
Understanding foreign markets
environment 12.471 122 0.000 0.9593 0.8071 1.1116
Resources required to working
globally 12.471 122 0.000 0.9593 0.8071 1.1116
Multilanguage web site 5.930 122 0.000 0.5528 0.3683 0.7374
Culture considerations 8.317 122 0.000 0.7317 0.5575 0.9059
Availability of international
delivery 7.020 122 0.000 0.6667 0.4787 0.8547
Technology infrastructure 6.687 122 0.000 0.6504 0.4579 0.8430
Internal culture 6.019 122 0.000 0.5772 0.3874 0.7671
Effective relationships between
the marketing and IS/MIS 6.774 122 0.000 0.6992 0.4949 0.9035
The great role of the sales force 4.919 122 0.000 0.4797 0.2866 0.6727
Training programs for sales force 4.222 122 0.000 0.4390 0.2332 0.6449
Trust 8.667 122 0.000 0.7236 0.5583 0.8889
Security 9.163 122 0.000 0.9512 0.7457 1.1567
Relationship with customers and
suppliers 12.825 122 0.000 1.0407 0.8800 1.2013 Table VI.
Easy and affordable access to the One sample t test of
internet 16.257 122 0.000 1.1626 1.0210 1.3042 statistical significance of
Customer acceptance 17.401 122 0.000 1.2439 1.1024 1.3854 B2B IIM CSFs
BIJ In fact some may be classified as a strategic factors, which require a significant change
13,1/2 in the manner of which business is being done (Turban et al., 1999). These include
Category A. marketing strategy related factors, such as top management commitment,
developing an integrated internet marketing strategy and setting the strategic goals for
internet use. This strategy will, in turn affect web site policy about accuracy,
information and design as well as ease of use. It will also determine external factors
210 like customer and supplier relationships, the nature of internet access in terms of
affordability and appropriateness. In general, all these factors should be considered at
the early and long-term planning phase of B2B IIM implementation.

Discussion of findings: tactical factors

The next group of factors can be classified as tactical. These include marketing
strategy related factors such as deciding the potential audience and the nature of
collaboration with strategic partners; WSF like effective online marketing of the site;
and global factors in terms of understanding of foreign marketing environment, the
resources required to work globally and the use of multi-language web sites. Tactical
factors might include internal considerations such as effective relationships between
the marketing and IS/MIS, the required technology infrastructure and internal culture,
as well as external issues like trust, security and culture.
However, we were surprised to find that “real time online interactive elements” has
shown only a negligible importance. But, upon closer examination of our study, this
finding relates to work carried out by Chaffy et al. (2000) who note that interactive
elements are important for B2C rather than B2B internet marketing.

Conclusions
This research was able to meet its objectives firstly: by identifying CSF for B2B IIM
and secondly: by empirically testing these factors using a large-scale survey of UK
companies in five different industrial sectors. Thirdly: it was able to develop a
framework for analysis using five dimensions (Categories A, B, C, D, E), and 33 factors
that can be adopted as a systematic measuring process, from a benchmarking
perspective, for those companies who want to make effective use of B2B IIM.
In fact this study provides new theoretical grounds for research into global B2B
internet marketing, as well as a guide for practitioner decision-making if they are to
remain competitive in the dynamic electronic marketplace. It provides an empirical
assessment of the essential elements in B2B IIM adoption, implementation, and
implications as well as a means to prioritise CSF. These have been distilled from a
comprehensive review of the relevant literature, and tested on marketing and IT
managers in 123 UK companies in aerospace, agriculture, chemical, computing,
industrial supplies and textiles.
Moreover, the CSF serve as building blocks for the development of a holistic
conceptual theory dealing with B2B internet use. They provide a framework for
academic theorists and highlight the significant role of marketing practitioners who
can play a significant part in developing and sustaining effective internet practice in
the global market place. Company managers can derive a better understanding and
measurement of activities that are undertaken by marketing professional within their
organisations, and derive an appropriate balance between traditional and internet
marketing practice. It also points to the need for future research to refine and validate
the framework as a benchmarking tool in the B2B, B2C and B2A (administration) International
environments, so that comparison can be made. At the same time the quantification of internet
CSF can be developed into an “index of practice” for companies to determine the level
of performance on a year-on-year time-based approach, for benchmarking purposes. marketing
In short, this study provides a thorough analysis of critical factors that enhance or
detract in current B2B IIM practice. It adopts a balanced perspective that incorporates
the dynamics of new technological opportunities, a volatile, fragmented global market 211
place, and the networks and relationships between company, customer, suppliers and
distributors. It addresses the challenges faced by many companies today in their
efforts to manage, control and advance daily business activities and find leverage
points within the world of B2B internet marketing.

References
Alshawi, S., Irani, Z. and Baldwin, L. (2003), “Benchmarking information technology investment
and benefits extraction”, Benchmarking: An international Journal, Vol. 10 No. 4, pp. 414-23.
Anderson, T. (1996), “Marketing products to the global desktop”, The Internet Strategy
Handbook: Lessons from the New Frontier of Business, Harvard Business School Press,
Boston, MA.
Avlonitis, G.J. and Karayanni, D.A. (2000), “The impact of internet use on business-to-business
marketing: examples from American and European companies”, Industrial Marketing
Management, Vol. 29, pp. 441-59.
Bremer, M. (1996), “Productivity, policy, and internet training issues”, The Internet Strategy
Handbook: Lessons from the New Frontier of Business, Harvard Business School Press,
Boston, MA.
Bryman, A. (1989), Research Methods and Organization Studies, Unwin Hyman, London.
Chan, C. and Swatman, P.M.C. (2000), “From EDI to internet commerce: the BHL steel
experiences”, Internet Research: Electronic Networking Applications and Policy, Vol. 10
No. 1, pp. 72-82.
Clarke, I. and Flaherty, T.B. (2003), “Web-based B2B portals”, Industrial Marketing
Management, Vol. 32 No. 1, pp. 15-23.
Chaffy, D., Mayer, R., Johnston, K. and Ellis-Chadwick, F. (2000), Internet Marketing,
Prentice-Hall, London.
Cronin, M.J. (1996a), The Internet Strategy Handbook: Lessons from the New Frontier of Business,
Harvard Business School Press, Boston, MA.
Damanpour, F. (2001), “E-business e-commerce evolution: perspective and strategy”, Managerial
Finance, Vol. 27 No. 7, pp. 17-33.
Duggan, M. and Devenery, J. (2000), “How to make internet marketing simple”, Communication
World, Vol. 17 No. 4, pp. 58-61.
Eid, R., Trueman, M. and Ahmed, A. (2002), “A cross-industry review of B2B critical success
factors”, Internet Research: Electronic Networking Applications and Policy, Vol. 12 No. 2,
pp. 110-23.
Furnell, S.M. and Karweni, T. (1999), “Security implications of electronic commerce: a survey of
consumers and businesses”, Internet Research: Electronic Networking Applications and
Policy, Vol. 9 No. 5, pp. 372-82.
Gogan, J.L. (1997), “The web’s impact on selling techniques: historical perspective and early
observations”, International Journal of Electronic Commerce, Vol. 1 No. 2, pp. 89-108.
BIJ Gurau, C., Ranchhod, A. and Hackney, R. (2001), “Internet transactions and physical logistics:
conflict or complementarity”, Logistic Information Management, Vol. 14 Nos 1/2, pp. 33-43.
13,1/2
Hair, J., Ralph, A. and Ronald, T. (1998), Multivariate Data Analysis, 5th ed., Prentice-Hall,
London.
Hamill, J. and Gregory, K. (1997), “Internet marketing in the internationalisation of UK SME’s”,
Journal of Marketing Management, Vol. 13, pp. 9-28.
212 Herbig, P. and Hale, B. (1997), “Internet: the marketing challenge of the twentieth century”,
Internet Research Electronic: Networking Application and Policy, Vol. 7 No. 2, pp. 95-100.
Hofacker, C.F. (2001), Internet Marketing, 3rd ed., Wiley, New York, NY.
Honeycutt, E.D., Flaherty, T.B. and Benassi, K. (1998), “Marketing industrial products on the
internet”, Industrial Marketing Management, Vol. 27 No. 1, pp. 63-72.
Huff, S.L., Koltermann, D. and Glista, J. (1998), “Settlers not surfers”, Ivey Business Quarterly,
Vol. 62 No. 4, pp. 45-9.
Karayanni, D.A. and Baltas, G.A. (2003), “Web site characteristics and business performance:
some evidence from international business-to-business organisations”, Marketing
Intelligence & Planning, Vol. 21 No. 2, pp. 105-14.
Kolokotronis, N., Margaritis, C., Papadopoulou, P., Kanellis, P. and Martakos, D. (2002), “An
integrated approach for securing electronic transactions over the web”, Benchmarking: An
International Journal, Vol. 9 No. 2, pp. 166-81.
Kotab, M. and Helsen, K. (2000), Global Marketing Management, 2nd ed., Wiley, New York, NY.
Lynn, G.S., Lipp, S.M., Akgun, A.E. and Jr A, C. (2002), “Factors impacting the adoption and
effectiveness of the world wide web in marketing”, Industrial Marketing Management,
Vol. 31 No. 1, pp. 35-49.
McGaughey, R.E. (2002), “Benchmarking business-to-business electronic commerce”,
Benchmarking: An International Journal, Vol. 9 No. 5, pp. 471-84.
Merrilees, B. and Fry, M. (2003), “The influence of perceived interactivity on e-retailing users”,
Marketing Intelligence & Planning, Vol. 21 No. 2, pp. 123-8.
Naude, P. and Holland, C. (1996), “Business-to-business relationships”, Relationship Marketing,
Theory and Practice, Paul Chapman Publishing Ltd, London.
Nunnally, J. (1978), Psychometric Theory, Mc Graw-Hill, New York, NY.
Perry, M. and Bodkin, C.D. (2002), “Fortune 500 manufacturer web sites: innovative marketing
strategies or cyberbrochures”, Industrial Marketing Management, Vol. 31, pp. 133-44.
Porter, M.E. (2001), “Strategy and the internet”, Harvard Business Review, Vol. 79 No. 3, pp. 63-78.
Poon, S. and Jevons, C. (1997), “Internet enabled international marketing: a small business
network perspective”, Journal of Marketing Management, Vol. 13, pp. 29-42.
Quelch, J.A. and Klein, L.R. (1996), “The internet and international marketing”, Sloan
Management Review, pp. 60-75.
Ratnasingham, P. (1998), “The importance of trust in electronic commerce”, Internet Research:
Electronic Networking Applications and Policy, Vol. 8 No. 4, pp. 313-21.
Rich, G.A. (2002), “The internet: boom or bust to sales organisations?”, Journal of Marketing
Management, Vol. 18 Nos 3/4, pp. 287-300.
Samiee, S. (1998), “Exporting and the internet: a conceptual perspective”, International Marketing
Review, Vol. 15 No. 5, pp. 413-26.
Shen, Q., Li, H., Shen, L., Drew, D. and Chung, J. (2003), “Benchmarking the use of information International
technology by the quantity surveying profession”, Benchmarking: An International
Journal, Vol. 10 No. 6, pp. 581-96. internet
Turban, E., McLean, E. and Wetherbe, J. (1999), Information Technology for Management: marketing
Making Connections for Strategic Advantage, Wiley, New York, NY.
Urban, G.L. et al. (2000), “Placing trust at the centre of your internet strategy”, Sloan
Management Review, Vol. 42 No. 1, pp. 39-48. 213
White, G.K. (1997), “International on line marketing of foods to US consumers”, International
Marketing Review, Vol. 14 No. 5, pp. 367-84.
Wilson, S.G. and Abel, I. (2002), “So you want to get involved in e-commerce”, Industrial
Marketing Management, Vol. 31 No. 2, pp. 85-94.
Zairi, M. (1998), Effective Management of Benchmarking Projects – Practical Guidelines and
Examples of Best Practice, Butterworth-Heinemann, Oxford.
Zairi, M. (2001a), E-Capability: From Transactional to Collaborative Competitiveness, Bradford
University, Bradford, Special presentation in Bradford School of Management.
Zairi, M. (2001b), From Craddle to Maturity: The Effective Management of Value Chains,
Inaugural Annual Lecture, Huddersfield University, Huddersfield, March.

Further reading
Bennet, R. (1997), “Export marketing and the internet: experience of web site use and perceptions
of export barriers among UK businesses”, International Marketing Review, Vol. 14 No. 5,
pp. 324-44.
Cronin, M.J. (1996b), Global Advantage on the Internet, Van Nostrand Reinhold, New York, NY.
Hoffman, D.L. et al. (1999), “Building consumer trust online”, Communication of the ACM, Vol. 42
No. 4, pp. 80-5.
Kiani, G.R. (1999), “Business-to-business advertising effectiveness in the web context”,
unpublished Phd thesis, Brunel University/Henley Management College,
Henley-on-Thames.
Kumar, V. (2000), International Marketing Research, Prentice-Hall, Upper Saddle River, NJ.
Shama, A. (2001), “E-coms and their marketing strategies”, Business Horizons,
(September-October), pp. 14-20.

Corresponding author
A.M. Ahmed can be contacted at: a.m.ahmed1@bradford.ac.uk

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints
 The current issue and full text archive of this journal is available at
www.emeraldinsight.com/1463-5771.htm

BIJ
13,1/2 Comparative evaluation of
practices: lessons from R&D
organizations
214
N. Mrinalini and Pradosh Nath
National Institute of Science, Technology and Development Studies,
New Delhi, India

Abstract
Purpose – To highlight a few critical issues related to indicators that are used to compare
organizational practices.
Design/methodology/approach – The approach is as follows: to compare the efficacy of
qualitative and quantitative indicators; objectivity in identification of processes and practices; and
make a distinction between essence and logistics of benchmarking.
Findings – For any benchmarking exercise the most important job is objectivity in identification of
processes and practices. The distinction between the two is not always apparent. Qualitative
evaluation of practices is as important as quantitative evaluation. The quantitative performance
indicators for evaluation of best practices come only at the end of a series of critical qualitative
analysis of the organizational processes and practices.
Research limitations/implications – The subject of the paper has been the enhancement of the
effectiveness of R&D organizations. Such organizations form an important part of the innovation
system of a nation, generally known as national innovation system (NIS). NISs of more successful
countries are being continuously evaluated and emulated by other countries for attaining respective
technological aspirations. This leaves a wide scope for application of benchmarking methodology. The
dynamics of networks like NIS being different from that of a business organization, the methodology
for best practices has to be refined. The present paper has addressed only the R&D organization part
of NIS. Much of the validity of the conclusions will depend upon studies on NIS in particular and
organizations not driven by profit motive in general.
Practical implications – The exact definition of the organization in terms of its mandate, long-term
vision and source of competence is the critical task. This is an important lesson for extending
benchmarking exercise beyond business enterprises. The study envisages that the methodology
would have wide scope of application for organizational restructuring of R&D organizations.
However, methodology has to be refined depending on the specificity of the problem of R&D
organizations.
Originality/value – It is in presenting the comparative efficacy of qualitative and quantitative
indicators for R&D organizations and also in highlighting the importance of benchmarking, the
essence of organizational practices, to arrive at the logistics.
Keywords Performance measures, Benchmarking, Research and development, Quality indicators,
Quantitative methods
Paper type Research paper

Benchmarking: An International
Journal
Vol. 13 No. 1/2, 2006
pp. 214-223
q Emerald Group Publishing Limited
1463-5771
DOI 10.1108/14635770610644691
Introduction
The concept of benchmarking is a simple exercise to target and emulate the best. The
task involves certain yardsticks to compare one with the other, to arrive at certain
comparable measures. The basis of comparison has to be some indices that will
measure the relative efficiency of relevant practices. Much of the simplicity of the
concept of benchmarking, however, is lost in the wearying job of objective delineation
of practices and comparing practices for identification of best practices. The present Comparative
paper intends to highlight some of the issues related to objective identification of the
practices to be benchmarked and also use of appropriate performance indicators (PIs)
evaluation of
for the exercise of benchmarking. The three issues that are discussed in the paper are: practices
(1) comparative efficacy of qualitative and quantitative indicators;
(2) objective identification of the practices for benchmarking; and
215
(3) benchmarking of essence and logistics of benchmarking.

The mainstay of the paper is a global study on non-corporate (government promoted

and funded) R&D organizations coordinated by World Association of Industrial
Technology Research Organizations (WAITRO) (WAITRO, 1996). Such R&D
organizations have been created to augment the technological strength of the
respective national enterprises. Profit maximization, therefore, has not been the
objectives of these organizations. WAITRO exercise thus differs from the widely
reported and well-discussed benchmarking exercise of business enterprises (Adler and
Cole, 1993; Camp, 1989; Bemowaski, 1991; Singh and Evans, 1993). This difference has
implications on the PIs that have been used in the WAITRO exercise. The annual profit
of an organization could not be used as the reference point for comparative evaluation
of the practices. It is the search for an alternative reference point that led to the issues
mentioned above. Only other study on non-corporate research organizations had been
reported by Rush et al. (1995). They, however, assumed an organization (and, therefore,
its practices) as best, and compared the practices of other organizations with the best
practices of the best organizations. We, on the contrary, did not begin with any given
set of best practices, nor the idea that the best organization would have all practices as
best practices.
Being very specific to the non-profit oriented R&D organizations, the exercise may
not appear to be directly relevant to the general purpose of benchmarking exercises for
business organizations. The paper, however, serves two purposes. First, it makes a
case for wider use of the technique, in addition to the business enterprises, and second,
it also raises some general issues that could be relevant for methodological refinement
of benchmarking technique.
The paper is divided as the following sections. In the section, organization of R&D,
we describe certain special characteristics that make R&D organizations different from
any other business enterprises. In the section, PIs, we argue for PIs specific to the
problems related to R&D organizations. The focus of the argument here is macro vs
micro level indicator and qualitative vs quantitative indicators. In the next two
sub-sections, we take up two issues related to R&D organization and examine them in
the light of our discussion on PIs. The two issues are: R&D organization as knowledge
generating organization; and R&D organization as learning organization. The final
section delineates the lessons drawn for the conclusion of the study.

Organization of R&D
Our focus is on non-corporate R&D organization. They have been created and
supported by respective national government for providing technological edge to the
domestic enterprises. Such R&D organizations have undergone many structural
changes, some planned and some as response to changing environment, over the last
four decades (Araoz, 1996, 1999; Bell, 1993). Many of these R&D organizations, both in
BIJ developed and developing countries, were facing the problem of adjustment with the
13,1/2 changing environment and expectations from them. Uncertain government support
added to that confusion (Nath and Mrinalini, 2002). While there are quite a few studies
that examined the nature of the problem faced by these organizations, WAITRO study
had aimed at understanding the organizational changes that would revitalise these
organizations. Identification and subsequent adoption of best organizational practices
216 had been thought to be the way out.
It is to be noted that we are dealing with R&D organization that is different from the
in-house R&D set of any enterprise or corporate sector. The organization, we are
referring to, is located outside the boundary of the corporate world, and at the same
time the corporate sector is the best client for them. These organizations are not driven
by profit motive, but supposed to augment the profit drive of the corporate sector, or
more appropriately the production sector of the economy. Decades before, when these
organizations were functioning under assured government support, many of these
organizations became reluctant partner of the production system. In the present day of
dwindling government support, they have to finely balance their activities between
generation of knowledge and revenue. In this act, to retain the legitimacy of their
existence, their activities have to be such that the knowledge they offer is not the part
of the market knowledge.
We shall, therefore, define R&D organizations as knowledge generating
organizations that also generates revenue for its sustenance in the process of
disseminating the knowledge. Once we define the domain of R&D organization in this
way, the evaluation criteria of their performance are set against their effectiveness in
the process of knowledge generation. So, the PIs should be able to compare, evaluate
and measure their knowledge generation capabilities. Again being defined as
organizations for generation of knowledge, a distinction between human resource
embodied and physical resource embodied knowledge becomes imperative. While both
human embodied and physical resource embodied knowledge are complementary to
each other, the former holds the key for distinctive identity of an R&D organization.
For the purpose of this paper, we shall present two cases related to human resource
issues of R&D organizations. Before that we present a brief understanding of the PIs
for arriving at best practices in this regard.

Performance indicators
In the case of organizational benchmarking, the task begins with categorisation of
organizational core activities in terms of processes and sub-processes. This
categorisation is basically to identify the decision-making points for action, at
process and sub-process level. These are nothing but the organizational practices[1].
Since a process or sub-process is constituted of a set of functionally related practices,
more and more detailing out of the processes into sub-processes would miss out the
nuance of functionality of practices, which make processes dynamic (Nath and
Mrinalini, 2000). The hard task of the PIs begins after the categorisation process is
accomplished. To be able to deliver a precise answer to the question of best practices
some sort of quantification would be necessary for measuring the performance of
practices under a particular process. What we need, therefore, is process-wise PIs so
that practices can be compared with each other and best practice can be benchmarked.
Three fundamental problems arise at this stage. Is it appropriate to have macro level
indicators for all processes or sub-processes? By macro level indicator we mean Comparative
indicators that reflect the performance of the organization as a whole and not specific evaluation of
to any particular process or sub-process. What would be the right PI and how can we
arrive at the appropriate indicator? Is quantitative indicator the best bet or qualitative practices
indicators can serve the purpose better in some occasions? This paper tries to throw
some light on these issues from the experience of the benchmarking study on R&D
organizations. 217
PIs are to be constructed so that they approximate a functional relationship with the
practices of a process that is to be benchmarked. General tendency is to construct
indicators that are based upon easily available, and usable macro level quantitative
data. In any organization, balance sheet based information is one such set of
quantitative data. It is generally agreed that benchmarking is not based on totally
factual and precise data. It is suggested that benchmarking exercise must rely on
imperfect data, or actionable accuracy. According to Sprow (1993), we benchmark the
most important, not the easiest to measure.
When we are constructing functionality between indicators and a process, we are
also assuming a continuous and differentiable function. In reality neither of these
properties can be imposed on organizational processes. Suppose when we rank
practices in terms of some quantifiable indicators, and say that, practice “A” score 5
per cent more than practice “B”, will that be sufficient for accepting or rejecting any
practice as good or bad?
A practice is a micro level action in an organization. The performance of an
organization is the cumulative effect of several such practices working in concurrence
with each other. The performance gets reflected in the form of certain macro level
financial balance sheet data. This is not practice-specific and reflects macro level
performance of the organization. Revenue or similar information, therefore, is the result
of simultaneous functioning of many practices involving many organizational
processes. To estimate the contribution of one particular practice on any organization
level data, one has to control the effect of other practices on the same. Unless that is
achieved, it would be inappropriate to choose one practice as an explanatory action
only on the basis of practice wise functional relations. Besides this, many processes
and practices may actually be passive and decorative. A macro level PI may, therefore,
over estimate its contribution. It is argued here with the case study of R&D
organizations that in case of organizational benchmarking, a set of qualitative
indicators may be more efficient in evaluating best practices than macro level
quantitative information like balance sheet based data.
We shall elaborate this argument with the two cases presented below. The first case
brings out the conflict between macro and micro level priorities of an organization and
exemplifies the need for micro level PIs. This case also indicates the need for
qualitative indicator over and above quantitative indicators. The second case relates to
the question of learning organization that nurtures and uses the human resources of an
organization.

Knowledge vs revenue maximisation

Amidst pressure for attaining self-sustainability, RTOs world over has reorganized
their activities with emphasis on linking up R&D activities with the production system
of the clients. Such linkages are aimed for development and transfer of R&D output to
BIJ the production system. The macro level PI for the best practices could be the
13,1/2 contributions of various practices to revenue, and thereby self-sustainability of the
organization. While revenue-related data of an organization is easier to gather and also
easier to analyse, difficulty, however, arises relating various practices to revenue
earning. Many of the organizational processes and practices are intended to facilitate
the uncertain and unknown tract of knowledge generation and possibly contribute to
218 revenue in an untraceable or at best in an extremely roundabout way. How do we
evaluate such practices?
The WAITRO (1996) study came across a few cases where R&D organizations
actually became self-sufficient over a very short period of time. When their
practices were studied, it has been revealed that the practice they had adopted had
strong long-term negative implications. The RTOs in question were actually
offering quite a few packages of technical services to their clients. Most of these
services were in competition with many other organizations dedicated to provide
such services. RTOs initially went for substantial price cut for its services to be
able to compete with the other organizations. RTOs also invested heavily for
acquiring sophisticated machines, etc. In fact, most of these services are
instrument or machine based. RTO has been able to capture a good share of the
market essentially because of its reputation as a research institution. The result
has been substantial increase in the revenue earning.
Is it the right practice towards attaining self-sustainability? If we look at the
practice more closely, it is apparent that the source of revenue has been knowledge that
is embodied in physical resources, or in machines and instrument. This is knowledge
that any organizations that acquire such physical resources can possess. In fact, a close
scrutiny of the activities of the RTO revealed that it has been systematically reducing
R&D activities for more revenue generating activities. The result was loss of human
resources in two ways. Many qualified and experienced R&D personnel left the
organization, and in fact many of them found opportunities in the clients’ enterprises.
What was gain for the clients was actually loss for the RTO. In the absence of R&D
projects there has been no human embodied knowledge generation, and hence no
human resource generation and utilisation. Gradually, the RTO lost its legitimacy as
R&D organization and became a full time technical service provider. Even in that it
will lose credibility because, as we mentioned earlier, much of its market credibility has
actually been drawn from its earlier identity as an R&D organization.
To arrive at this understanding the evaluation of practices had to be done with a
different set of criteria. First of all, RTOs were to be defined more carefully not as a
revenue maximising organization, but as knowledge generating organization that
earns substantial revenue in the process. Given this definition, the PIs for its human
resource related processes could not be revenue generation but knowledge generation.
Again instead of looking for quantifiable indicator for knowledge generation, some sort
of qualitative assessment of RTO activities has been made. Basic question has been
that if the RTO is doing something or having capabilities that others in the market do
not have. This question is justified because the RTO is mandated to generate and
disseminate knowledge. If it engages itself in the activities that many others can do, the
need for the RTO would be in question. Again the very act of developing activities
around physical resources reducing the role of human resource is detrimental for the
long-term survival of the RTO as R&D organization (Mrinalini and Nath, 2000).
 It is to be noted here that some sort of quantification is possible between relative Comparative
importance of human resources and physical resources in the RTO. The point we evaluation of
wanted to put forward with this example is that even if some quantifiable indicator is
possible to generate, it is to be done at the micro level, or in the present context at the practices
sub-process level. In this case, it is for the sub-process “human resources”. We would
also like to point out that we have arrived at the possible quantifiable indicator only
after qualitative understanding of the basic principles of the process and its function. 219
Such quantification, however, could be misleading if its qualitative implications are not
fully understood. Let us try to clarify the argument further. In any R&D organization
both human and physical resources play complementary roles. Suppose we take
balance sheet data for invest on human resources and physical resources. We observe
that there is high expenditure on human resources compared to that in the physical
resources. Now, going by our previous argument do we say that the practice is a good
practice? On the contrary such a practice could be worst practice for an R&D
organization, where there is substantial human resources but inadequate physical
resources to complement the R&D activities.
To look for the best practice, therefore, we have to see how best the human and
physical resources complement each other. Accordingly, we have to look for
appropriate proxies that can capture real situation. What we, therefore see, is that lot of
qualitative analyses have to precede a quantitative indicator. Such indicators if at all
found, will be for answers to very specific question related to a practice or process, and
cannot be used at aggregate level.

Human resource and learning organization

Let us take a simple definition of learning organization. Organization that has
arrangements for learning by its people and also for making the individual learning
available for the organization can be called a learning organization. Question is what
are the practices that can ensure making a learning organization?
Let us go back to the question of RTO best practices. A RTO, engaged in industrial
technology research, is not for generation of knowledge in isolation, nor it is for
dissemination of common knowledge that will maximize its revenue in the short-run. It
has to be responsive to clients’ demand by exposing them to and providing them with
RTO services based on higher level of knowledge. Two most important components of
this task are to become responsive to clients needs and demands, and to be able to
access and process higher level of knowledge to provide technological services to
clients. RTOs’ internal organization has to be structured in such a way that it can work
as the link between the knowledge world and the clients’ world – the production
system (Nath and Mrinalini, 2002). What are the basic organizational principles behind
the efficiency of RTO internal organization? Once such principles are identified,
different organizational processes and practices of a RTO have to be examined in that
light.
Understanding of a learning organization provides such principles. While
evaluating RTO practices, we have to see if those practices help attaining the
characteristics of a learning organization. Internal organization of a RTO has to be
such that the human embodied knowledge that is acquired by the individuals of the
organization is available for the organization. A highly autocratic organizational
structure may alienate the human resources to such an extent that organization might
BIJ not be able to benefit from the knowledge possessed by its own manpower. A
13,1/2 bureaucratic organization, on the other hand, may restrict the use of knowledge
resources to routine decision-making (Choo, 1996). In a flat organization, people may
not have enough incentive to make their knowledge available to the organization in the
absence of power and authority associated with performance. Dovey (1997), discusses
the organizational structure with community values (group, team). The community
220 value based organizational structure is quite abstract and to a great extent idealistic.
Besides, there is no mechanism in the system to ensure that every member in the
community shares the same values to the same extent. In fact, the
socio-anthropological evidences on community dynamics indicate that such
structures are actually highly autocratic in the pursuits of community values.
The WAITRO study has come across the internal organizational structure of RTOs
that varied from highly autocratic to bureaucratic to flat organizational structure. Such
structures could not be directly related to the performances of the RTOs. From the
analysis of the various practices what transpired was that individual’s commitment to
the organization greatly depends, among other things, on the empowerment along the
hierarchy. Again empowerment is counterbalanced by accountability. The
empowerment does not have to be uniform across the organizational hierarchy. In
fact, variation in the degrees of empowerment and associated accountability creates
identity of an individual in an organization and breaks the alienation between
individual and the organization. Again empowerment is the motivating factor for
individuals to give the best for the organization. It also works for infusing team culture
or in a broad sense community culture of functioning within the organization. While
the known organizational structures (bureaucratic, autocratic, etc.) can be
distinguished in terms of type, extent and nature of empowerment, it is quite
possible to have a flat organization with various degrees of empowerment.
Given the above understanding what do we benchmark and what would be the
indicators? Clearly, we benchmark the degree of empowerment and accountability in
the organization. We also evaluate how empowerment varies across the organization.
In one of the RTOs there has been the practice of empowerment by involving
individual staff in various committees that are constituted for taking decisions
regarding organizational matters. In such a system, everybody is empowered for
taking some or other decisions. With the empowerment, is associated accountability
for the consequences of the decisions taken by a committee. All the committee
members have to endorse a decision and so have to defend. This creates a community
feeling.
It is to be noted here that the evaluation of the practices in this case has been totally
qualitative, but based on certain precise understanding of organizational needs and
dynamics. In fact, in such cases it is difficult to think of any quantitative indicator that
would better our present understanding of the good and bad practices.

Lessons drawn
The cases presented here do not answer the questions like, what should be the right
proportion of investment for human and physical resources, or what is the degree of
empowerment that would ensure the formation of a learning organization? What
actually has been benchmarked here is the essence of a practice. We arrive at the
understanding that better practice is when human resource is given priority and
complemented adequately with physical resources. Also empowerment helps Comparative
development of community feeling within the organization. We are still away from evaluation of
the actual logistics of the practice. There could be various ways of achieving the
desired result. Much of it would evolve through experimentation within the practices
organization. Since an organization is a socio cultural entity, such experimentation has
to incorporate the social and cultural parameters within which the organization has to
function. 221
We have made distinction between macro level and micro level indicators for
evaluation of the best practices. We have argued that macro level indicators have to be
essentially qualitative. Such qualitative evaluation helps us to reduce the issue to
actionable questions that can be quantitatively evaluated. It was qualitative evaluation
that helped us to identify that revenue generation alone was not enough for long-term
legitimacy of the R&D organization. Quantifiable indicators come to use to decide how
to allocate resources to have the right balance between human and physical resources.
The mainstay of the exercise, however, is the exact definition of the organization in
terms of its mandate, long-term vision and source of competence. The cases presented
here show how critical this task is. This is an important lesson for extending
benchmarking exercise beyond business enterprises.

Conclusion
With two examples, we have argued that quantitative PIs for evaluation of best
practices come only at the end of a series of critical qualitative analyses of the
processes and practices. Most of the qualitative analyses have to be based on certain
principles that precisely explains the dynamics of the practice and defines the
relationship between cause and effect. From the above discussion, it appears that in
any benchmarking exercise, the most important job is a comprehensive and
meaningful categorisation of the organization. Once that is achieved constructing
qualitative and quantitative indicators to approximate the best practice would not be
very difficult.
The subject of the paper has been the enhancement of the effectiveness of R&D
organizations. Such organizations form an important part of the innovation system of a
nation, generally known as national innovation system (NIS) as articulated by Nelson
(1993) and Lundvall (1992). NISs of more successful countries are being continuously
evaluated and emulated by other countries for attaining respective technological
aspirations. This leaves a wide scope for application of benchmarking methodology.
The dynamics of networks like NIS being different from that of a business
organization, the methodology for best practices has to be refined. The present paper
has addressed only the R&D organization a part of NIS. Much of the validity of the
conclusions will depend upon studies on NIS in particular and organizations not driven
by profit motive, in general.

Note
1. This paper is based on an International study “strengthening the capability of research and
technology organizations”, funded by IDRC, Canada and DANIDA Denmark and
coordinated by World Association of Industrial Technology Research Organizations
(WAITRO). Authors were part of the international research team for the project. The task
entrusted to the research team was to study the organizational practices adopted by various
BIJ non-corporate RTOs, world over and come out with a set of “best practices” for enhancing
RTO-Client interaction. The paper draws the issues related to PIs from the book
13,1/2 Organization of R&D: An Evaluation of Best Practices, by the authors Nath and Mrinalini
(2002).

References
222 Adler, P.S. and Cole, R.E. (1993), “Designed for learning: a tale of two auto plants”, Sloan
Management Review, Vol. 34 No. 3, pp. 85-94.
Araoz, A. (1996), “Revitalization of industrial technology research institutes in developing
countries”, in Quereshi, M.A. (Ed.), Human Resource Needs for Change in R&D Institutes,
WAITRO-NISTADS, New Delhi.
Araoz, A. (1999), “Best practices among scientific research institutes responding to strategic
challenges”, unpublished project report, IFIAS.
Bell, R.M. (1993), “Integrating R&D with industrial production and technical change:
strengthening linkages and changing structures”, paper presented at the UNESCO
Workshop on Integration of Science and Technology in Developing Planning and
Management Process.
Bemowaski, K. (1991), “The benchmarking bandwagon”, Quality Progress, January, pp. 19-24.
Camp, R.C. (1989), Benchmarking: The Search for Industry Best Practices that Lead to Superior
Performance, Quality Press, Milwaukee, WI.
Choo, C.W. (1996), “The knowing organization: how organizations use information to construct
meaning, create knowledge and making decisions”, Journal of Information Management,
Vol. 16 No. 5, pp. 329-40.
Dovey, K. (1997), “The learning organization and the organization of learning power,
transformation and the search for form in learning organization”, Management Learning,
Vol. 28 No. 3, pp. 331-49.
Lundvall, B-Å. (Ed.) (1992), National Systems of Innovation: Towards a Theory of Innovation and
Interactive Learning, Pinter Publishers, London.
Mrinalini, N. and Nath, P. (2000), “Organizational practices for generating human resources in
non-corporate research and technology organizations”, Journal of Intellectual Capital, Vol. 1
No. 2, pp. 177-86.
Nath, P. and Mrinalini, N. (2000), “Benchmarking the best practices of non-corporate R&D
organizations”, Benchmarking: An International Journal, Vol. 7 No. 2, pp. 86-97.
Nath, P. and Mrinalini, N. (2002), Organization of R&D: An Evaluation of Best Practices, Palgrave
Macmillan, New York, NY.
Nelson, R.R. (Ed.) (1993), National Innovation Systems: A Comparative Analysis, OUP, New York,
NY.
Rush, H., Hobday, M., Bessant, J. and Arnold, E. (1995), “Strategies for best practices in research
and technology institutes: an overview of a benchmarking exercise”, R&D Management,
January.
Singh, D.K. and Evans, R.P. (1993), “Effective benchmarking: taking the effective approach”,
Industrial Engineering, Vol. 25 No. 2, pp. 22-4.
Sprow, E.E. (1993), “Benchmarking: a tool for our time?”, Manufacturing Engineering, Vol. 111
No. 3, pp. 56-69.
WAITRO (1996), “Strengthening research and technology organization’s capabilities”,
unpublished project report, WAITRO, Copenhagen.
Further reading Comparative
Boxwell, R.J. (1994), Benchmarking for Competitive Advantage, McGraw-Hill, New York, NY. evaluation of
Sprow, E.E. (1994), Benchmarking Manufacturing Processes – Your Practical Guide for Becoming
Best in the Class, Society of Manufacturing Engineering, Dearborn, MI.
practices

Corresponding author 223

N. Mrinalini can be contacted at: nmrinalini@nistads.res.in

To purchase reprints of this article please e-mail: reprints@emeraldinsight.com

Or visit our web site for further details: www.emeraldinsight.com/reprints