HOW TO GET RESPONSES FOR MULTI-CRITERIA DECISIONS IN

ENGINEERING EDUCATIONAN AHP BASED APPROACH FOR

SELECTION OF MEASURING INSTRUMENT
K.G.Viswanadhan
Research Scholar, Department of Management Studies
Indian Institute of Science, Bangalore, India 560 012

Abstract
The Analytic Hierarchy Process (AHP) is a
mathematical technique for multi-criteria
decision-making. It enables people to make
decisions involving many kinds of concerns
including planning, setting priorities, selecting
the best among a number of alternatives, and
allocating resources. Complex problems or issues
involving value or subjective judgments are
suitable applications of the AHP approach.
Quality Issues of Undergraduate Engineering
Education, the research topic of interest,
involves a number of qualitative judgments based
on multi-criteria at multi-levels and can be
addressed using AHP. Expert opinions are
collected in AHP using pair wise comparisons. As
the number of alternatives increases, the number
of comparisons increases enormously. Hence the
selection of a proper measuring instrument that
will reduce the data collection efforts with out
loss of reliability of measurement is very
important. Five sets of questionnaires have been
developed for collecting the expert opinions. A
pilot study (questionnaire administration) has
been conducted among five faculties with entirely
different backgrounds and their responses are
analyzed for consistency. Later on their opinions
are collected about the five sets of questionnaires
based on three criteria easiness, clarity and
extraction of correct responses. AHP technique is
used to prioritize the opinions and finally in the
selection of the measuring instrument. The paper
discusses the AHP technique, research problem,
development of questionnaires, and the AHP
method used for the final selection of the
measuring instrument.
Introduction
Socio- Economic systems often face decision
making with qualitative and intangible factors.
Values, beliefs and perceptions are the force
behind many of these decision-making activities.

Decision making processes relies on the information of

alternatives. Information may vary from scientifically
derived hard data to subjective interpretations with
certainty about decision outcomes to uncertain
outcomes. More over the decision depends on multiple
criteria and might not be a straightforward one. The key
problem in this situation is to elicit systematic
judgments from unstructured information. Analytic
Hierarchy Process (AHP) is a powerful tool for this
purpose (B Liu & S Xu, 1987). Complex problems or
issues involving value or subjective judgments are
suitable applications of the AHP approach.
Analytic Hierarchy Process
AHP is a mathematical technique used for multi-criteria
decision-making. In a way it is better than other multicriteria techniques, as it is designed to incorporate
tangible as well as non-tangible factors especially where
the subjective judgments of different individuals
constitute an important part of decision making (Saaty,
1980). Apart from other facts, this is rooted in the
special structure of the AHP, which follows the intuitive
way in which managers solve problems, and in its easy
handling compared with other multi criteria decisionmaking procedures. Hence the intuitively solved
decision problems can now be solved as procedureorientated using AHP. The use of AHP leads to both,
more transparency of the quality of management
decisions and an increase in the importance of AHP
(Ossadnik W& Lange O, 1999).
Because of its intuitive appeal and flexibility, many
corporations and governments routinely use AHP for
making major policy decisions. Applications of AHP can
be seen in a wide range of areas like merit salary
recommendation system (Troutt M D & Tadisina S K,
1992), environmental impact assessment (Ramanathan
R, 2001), credit evaluation of the manufacturing firms
(Yurdakula M & Tansel Y, 2003), indoor environment
assessment (Chianga CM & Laib CM, 2002), selection
of alternative transportation options (Yedla S & Shrestha
RM, 2003), performance measurement system
(Suwignjo P et al, 2000), TQM implementation (Chin K
S et al, 2002), evaluation of highway transportation

(Weiwu W & Jun K, 1994), determination of key

capabilities of a firm (Hafeez K et al, 2002) and
for evaluation of an AHP software (Ossadnik W
& Lange O, 1999) itself.

their opinions and information as the key resource for

the study. Obviously the opinions will be subjective, and
based on multiple criteria. Hence AHP is adopted to
analyze the problem.

AHP uses a five-step process to solve decision

problems. They are

Several researchers in educational field have been using

AHP as a decision making tool. It is claimed that AHP
can be used to find out the extent to which a student
understands the objectives of the engineering exercise
being tackled and the relative merits of the alternative
solutions (Drake P. R, 1998). One paper has described
the use of AHP for curriculum design (Frair L et al,
1998). A model of the problem is developed using a
hierarchical representation and the combination of
various AHP inputs for the ranking of various
curriculum alternatives is discussed in the paper. An
AHP model and software to support the associated
analysis is developed for justification of excellence of
technical institutions and presented in another paper
(Kodali R, 1998). Another applications of AHP
published (Rao RV & Singh D, 2002; Reddy KB et al,
2004) are in the field of performance evaluation of
technical education institutions.

Create a decision hierarchy by breaking

down the problem into a hierarchy of
decision elements.
Collect input by a pair wise comparison
of decision elements.
Determine whether the input data satisfies
a consistency test. If it does not, go back
to Step 2 and redo the pair wise
comparisons.
Calculate the relative weights of the
decision elements.
Aggregate the relative weights to obtain
scores and hence rankings for the
decision alternatives.

One of the major reasons for the popularity of

AHP is that the decision maker does not
require advanced knowledge of either
mathematics or decision analysis to perform
first two steps (Karapetrovic S & Rosen
bloom ES, 1999). Last three steps are
computational and can be performed
manually or using software such as Expert
Choice. However, the first two are the steps
where the decision maker is very much
involved in the model. On the basis of the
decision maker's understanding of the
problem, the hierarchy can be designed and
pair wise comparisons can be made of the
decision elements. AHP uses redundant
judgments for checking consistency, and this
can exponentially increase the number of
judgments to be drawn out from decision
makers.
Research problem
Unprecedented growth in engineering education
institutes in India in the last decade led to
complex issues. The main issue was that of
maintaining the quality of education provided by
these institutes. Many reasons are pointed out as
the cause of this situation. A study has been
conducted to find out the quality issues of
undergraduate engineering education sector. As
the faculties of engineering colleges are the
experts and one of the main stakeholders of
engineering education, it is decided to collect

Most of these papers from the educational field are

theoretical works and not based on actual data collected
as a part of research work. Hence the present work is
framed so as to collect primary data from the faculty of
engineering colleges and use this information to
prioritize the quality issues of undergraduate engineering
education in India. The hierarchy designed for the
present research problem (given in appendix) includes,
two levels of decision-making. Six alternatives are to be
compared in the first level. The number of judgments for
comparing alternatives in the first level will be 15 (6c 2).
The second level contains a total of 19 alternatives
coming under the 6 alternatives of first level. Number
of comparisons at the second level will be 22 (4c 2+3c2
+3c2 +2c2 +3c2 +4c2). Hence the total number of pair wise
comparisons for the two levels will be 37. This is often a
tiring exercise for the decision-maker. Design of a
proper questionnaire for data collection is very
important in this situation. This paper discusses the
process of decision-making adopted for the selection of
the best questionnaire from a number of alternatives.
Objective
To develop the best questionnaire for data collection for
the pair wise comparison of AHP method of prioritizing
the quality issues of undergraduate engineering
programmes.
Methodology
Five types of questionnaires were developed for
collecting the pair wise comparison of quality issues of

engineering education. Three criteria were

considered while designing the questionnaire.
They are easiness to fill up the questionnaire,
clarity of questions and extraction of intended
responses. The structures of the questionnaires
with sample questions are given in the appendix.
AHP is selected as the method of decision
process.
Step 1: Formation of hierarchy
The decision hierarchy is formulated by breaking
down the problem into a hierarchy of decision
elements and given in figure 1.
Step 2: Collection of inputs
AHP is used as a tool for systematically analyzing
the opinions of several experts belonging to
diverse fields in this step. Consulting more
experts will avoid bias that may be present when
the judgments are considered from a single
expert. Hence five faculties from five engineering
colleges coming under five different states
(Gujarat, Andhra, Karnataka, Tamil Nadu and
Kerala) are selected as the experts for the
decision-making. The nominal-ratio scale of 1 to
9 (Saaty, 1994) is adopted for pair wise
comparison of the questionnaires.
All the five questionnaires were given randomly
to the experts and requested to fill up the
questions in all the five questionnaires. Then they
were asked to conduct a pair wise comparison of
the five questionnaires, and to rank them based on

the three criteria easiness, clarity, and extraction of

correct responses. All the five responses were collected
and recorded.
Step 3: Consistency Test
The results of pair wise comparisons are filled in
positive reciprocal matrices to calculate the eigenvector
and eigenvalue (table 1 to 5). The consistency of the
judgments is determined by a measure called
consistency ratio (C.R.). The consistency ratio is
obtained to filter out the inconsistent judgments, when
the value of the consistency index (C.I.) is greater than
0.1. All the judgments are found to be consistent and
accepted for analysis.
Steps 4&5: Calculation of relative weights & ranking
of alternatives
Geometric mean method has been the most widely
applied method in AHP for aggregation of individual
preferences when more than one expert is involved in
the decision-making. Though all the expert opinions are
consistent, the opinion of third expert seemed to be
entirely different from others. He is informed of this
variation and asked whether he wants to stick on to his
earlier opinion or ready to modify the same to match
with the others. The expert selects second option and
the pair wise comparisons are modified accordingly.
Geometric mean of individual opinions are calculated
and entered in the final judgmental matrix for finding
out the ranks of the alternatives. The judgmental matrix
and the ranks of the questionnaires are given in table 6.

Selection of best
Questionnaire

Easiness

Questionnaire1

Questionnaire2

Clarity

Questionnaire3

Extraction of correct
responses

Questionnaire4

Questionnaire5

Fig 1 Hierarchy of selection process

Results and Analysis

The priority vector indicates that Questionnaires 1
and 2 are equally good for data collection.
Questionnaire 4 is also acceptable to the experts.
Other options, especially the fifth one is totally
unacceptable to the experts. It is clear from the
rankings that a detailed and simple questionnaire is
preferred to a compact and less time consuming
method. Fifth questionnaire, though very simple
cannot gather the required information of pair wise
comparisons, and hence rejected by the experts.
Third questionnaire is very compact but a novice
user who is unaware of matrix structures will find it
very difficult to fill it up. Hence the experts while
taking decision neglect the easiness to the researcher
for collecting and storing data. Based on the rankings
given by the experts and through the discussions with
them, it is decided to develop a modified
questionnaire, which is a combination of first, second
and fourth types.
Table 1: Opinion of Expert 1

Q1
Q2
Q3
Q4
Q5

Q1
1
2
0.5
0.33
0.5

Q2 Q3 Q4 Q5
0.5 2
3
2
1
3
4
2
0.33 1 0.5 1
0.25 2
1 0.5
0.5 1
2
1

Table 2: Opinion of Expert 2

Q1
Q2
Q3
Q4
Q5

Q1
1
3
0.2
0.33
0.33

Q2 Q3 Q4 Q5
0.33 5
3
3
1
3
1
3
0.33 1 0.33 0.33
1
3
1
5
0.33 3 0.2 1

Table 3: Opinion of Expert 3

Q1
Q2
Q3
Q4
Q5

Q1 Q2 Q3 Q4 Q5
1
3
3 0.33 0.2
0.33 1 0.33 0.2 0.2
0.33 3
1 0.2 0.2
3
5
5
1 0.2
5
5
5
5
1

Conclusions

Today most of the decisions are to be taken in

increasingly complex environments. Most of them
require different value systems and the use of experts
from different fields. They succeed by using
knowledge that is imprecise rather than precise. AHP,
which is a transparent technique, is very useful to
handle this type of situations where qualitative data
is involved in the decision-making. The use of AHP
does not involve cumbersome mathematics. AHP
involves the principles of decomposition, pair wise
comparisons, and priority vector generation and
synthesis.
An application of AHP to a decision-making
problem from engineering education field is
described in this paper. AHP methodology for the
selection of an AHP-pair wise comparison
questionnaire is discussed with illustrations. Five
types of questionnaires are developed and analyzed
with respect to three criteria. More insights about the
essential features of a measuring instrument are
obtained and the analysis finally leads to the
development of a new structure to the measuring
instrument.
Table 4: Opinion of Expert 4
Q1

Q2

Q3

Q4 Q5
0.14
Q 1 1 0.2 5
3
9
Q2 5
1
5 0.33 9
0.14
Q 3 0.2 0.2 1
3
9
Q4 7
3
7
1
9
0.11 0.11 0.11 0.11
Q5 1
1
1
1
1
Table 5: Opinion of Expert 5

Q1
Q2
Q3
Q4
Q5

Q1 Q2 Q3 Q4 Q5
1
3
3
3
7
0.33 1
1
1
7
0.33 1
1
1
7
0.33 1
1
1
7
0.14 0.14 0.14 0.14
3
3
3
3
1

Table 6: judgmental matrix and the ranks of the

questionnaires
Q1 Q2 Q3 Q4 Q5
Priority Vector
Q 1 1 0.56 3.5 1.40 4.41
0.295
Q 2 1.78 1 2.6 1.07 4.41
0.295
Q 3 0.29 0.39 1 0.39 2.14
0.11
Q 4 0.71 0.93 2.55 1 3.54
0.24
Q 5 0.23 0.23 0.47 0.28 1
0.06

References
1. Chianga CM, Laib CM, 2002. A study on the
comprehensive
indicator
of
indoor
environment assessment for occupants
health in Taiwan.
Building and
Environment, 37.
2. Chin K S, Pun K S, Xu Y, Chan J S F, 2002.
An AHP based study of critical factors for
TQM
implementation
in
Shanghai
manufacturing industries. Technovation 22.
3. Drake P. R, 1998. Using the Analytic
Hierarchy Process in Engineering Education,
Int. J. Engineering Education. Vol. 14, No. 3.
4. Frair L, Matson J.O, Matson J.E, 1998. An
undergraduate curriculum evaluation with
the Analytic Hierarchy Process. Proceedings
of 1998 FIE Conference.
5. Hafeez
K,
Zhang Y, Malak
N,
2002.Determining key capabilities of a firm
using
analytic
hierarchy
process.
International
Journal
of
Production
Economics 76.
6. Karapetrovic S,. Rosenbloom E.S, 1999.A
quality control approach to consistency
paradoxes in AHP, European Journal of
Operational Research 119.
7. Kodali R, 1998. Multi-attribute decision
model using analytic hierarchy process for
the justification of excellence of technical
educational institutions in India. The Indian
journal of technical education, Vol.21, No.3.
8. Liu B , Xu S, 1987. Development of the
theory and methodology of the Analytic
Hierarchy Process and its applications in
China. Mathl. Modeling, Vol 9, No. 3-5.
9. Ossadnik W, Lange O, 1999. AHP-based
evaluation of AHP-Software. European
Journal of Operational Research 118.

10. Ramanathan R, 2001, A note on the use of

the analytic hierarchy process for
environmental impact assessment. Journal of
Environmental Management 63.
11. Rao RV, Singh D, 2002. Analytic Hierarchy
Process (AHP) for the performance
evaluation of technical institutions. The
Indian journal of technical education, Vol.25,
No.4.
12. Reddy KB, Ayachit NH, Venkatesha MK,
2004. A Theoretical method for performance
evaluation of technical institutions - Analytic
Hierarchy Process approach, The Indian
journal of technical education, Vol.27, No.1.
13. Saaty, T.L, 1980. The Analytic Hierarchy
Process, McGraw-Hill, New York, NY.
14. Saaty, T.L, 1994. Fundamentals of Decision
Making and Priority Theory with the
Analytic
Hierarchy
Process,
RWS
Publications, Pittsburgh, PA.
15. Suwignjo P, Bititci U.S, Carrie A.S, 2000.
Quantitative models for performance
measurement system. International Journal
of Production Economics 64.
16. Troutt M D, Tadisina S K, 1992. The analytic
hierarchy process as a model base for a merit
salary
recommendation
system.
Mathl.Comput. Modeling vol.16, no. 5.
17. Weiwu W, Jun K, 1994. Highway
transportation comprehensive evaluation.
Computers ind. Engng, VOL. 27, Nos. 1-4.
18. Yedla S, Suresh RM, 2003. Multi-criteria
approach for the selection of alternative
options for environmentally sustainable
transport system in Delhi. Transportation
Research, Part A37.
19. Yurdakula M, Tansel Y, 2003. AHP
approach in the credit evaluation of the
manufacturing firms in Turkey, Int. J.
Production Economics (Article in press).

Appendix
Structure of Quality issues of undergraduate engineering education in India model

Qual i t y i s s ues of
Under gr aduat e engi neer i ng educat i on

Goal 1

Fact or 1

Goal 2

Fact or 2

Fact or 3

Goaql 3

Fact or 4

Fact or 5

Fact or 6

yCt or 1
Sub Fact or 1
Sub Fact or 1
Sub Fact or 1
Sub Fact or 1
Sub Fact or 2
Sub Fact or 2
Sub Fact or 2
Sub Fact or 2
Sub Fact or 3
Sub Fact or 3
Sub Fact or 3
Sub Fact or 4

Sub Fact or 1
Sub Fact or 1
Sub Fact or 2
Sub Fact or 2
Sub Fact or 3
Sub Fact or 3
Sub Fact or 4

Questionnaires
Questionnaire 1: Prioritization of factors for Stakeholder Satisfaction

Mark the relative importance of Organization & Governance, Financial & Physical Resources, H R-Staff,
H R-Students, Teaching-Learning Process and Other Processes with respect to the Stakeholder Satisfaction
of your undergraduate engineering programme.
Please put tick mark at a point, which is most nearer to your opinion, in each of the following scales.
Scale 1

Organization & Governance isFinancial & Physical Resources is

Absolutely
Important

V.Strongly
Important

Strongly
Important

Slightly
Equally
Slightly
Strongly
V.Strongly
Absolutely
Important Important Important Important
Important
Important

ThanThan
Financial & Physical Resources

Scale 2

Organization & Governance

Organization & Governance is

Absolutely
Important

V.Strongly
Important

Strongly
Important

Human Resources-Staff is

Slightly
Equally
Slightly
Strongly
V.Strongly
Absolutely
Important Important Important Important
Important
Important

ThanThan
Human Resources-Staff

Organization & Governance

Questionnaire 2: Prioritization of factors for Stakeholder Satisfaction

Questionnaire 2: Prioritization of factors for Stakeholder Satisfaction

Mark your opinion about the relative importance of the factors given on the two sides of the scales
(refer the scoring pattern). Please put tick marks on the number of your choice on each scale.
Organization & Governance 9
Organization & Governance 9

Scoring pattern

7
7

3
1
V.Strongly
Important
B

3
3

LHS
A is

97 5
Absolutely
Important
Than

5
5

3 5
Strongly
Important

1
1

3
3

5
5

7
7

9 Financial & Physical Resources

9 HR-Staff

MIDDLE

RHS

B is
7
9
Slightly
Equally
Slightly
Strongly
V.Strongly
Absolutely
Important Important Important Important
Important
Important
Than
A

If A is more important, use left hand side (LHS) of the scale.

If A and B are equally important, put tick mark on center portion (MIDDLE) of the scale.
If B is more important, use right hand side (RHS) of the scale.
Factors
FPR
HRS
HRSs
Organization & Governance (OG)
Financial & Physical Resources (FPR)
Human Resources - Staff (HRS)
Human Resources - Students (HRSs)
Teaching learning Processes (TLP)
Other Process (OP)
Questionnaire 3: Prioritization of factors for Stakeholder Satisfaction

TLP

OP

Mark the relative importance of Organization & Governance, Financial & Physical Resources, HR-Staff,
HR-Students, Teaching-Learning Process and Other Processes for Stakeholder Satisfaction
of your undergraduate engineering programme using the following scoring pattern.
Scoring Pattern
Relative Importance (More)
Equal importance
Slightly more important
Strongly more important
Very Strongly more important
Absolutely more important
Intermediate values

Score
1
3
5
7
9
2,4,6,8

Relative Importance (Less)

Equal importance
Slightly less important
Strongly less important
Very Strongly less important
Absolutely less important

Please fill up the white space provided in the matrix

Score
1
1/3
1/5
1/7
1/9
1/2,1/4,1/6,1/8

Questionnaire 4: Prioritization of factors for Stakeholder Satisfaction

Among each pair of given factors, which is more important for Stakeholder Satisfaction of your
undergraduate engineering programme and how much more important?
Put tick marks in the space provided.
1. Organization & Governance

Financial & Physical Resources

How much more important?

Equally

slightly more

2. Organization & Governance

strongly more

V.Strongly more

absolutely more

Human Resources-Staff

How much more important?

Equally

slightly more

strongly more

V.Strongly more

absolutely more

Questionnaire 5: Prioritization of factors for Stakeholder Satisfaction

What is your opinion about the importance of the following factors with respect to the
achievement of Stakeholder Satisfaction by your undergraduate engineering programme?
Use a 1 to 9 scale (1- least important, 9- most important and 2 to 8 for in between values).
Please put tick mark on the number of your choice on each scale.
Organization & Governance

Financial& Physical Resources 1

Human Resources-Staff