Comparative study of DTMF and ASR ISR system
Omkar Yadav Mahesh Singh
T.E.Comps T.E.Comps
SPIT,Andheri SPIT,Andheri
omkarr27@gmail.com Mahesh_spit@rediffmail.com
ABSTRACT:
This is a review paper of the paper ‘A Comparison
between DTMF and ASR IVR Services through Objective
and Subjective Evaluation’ by
Cristina Delogu, Andrea Di Carlo, Paolo Rotundi, Danilo
Sartori which examines the comparisons between
various types of ASR services and DTMF in the
context of IVR applications used in the modern call
centres. The users of this service utilise this
application for activating a particular scheme or
getting information about any other things related to
their network service provider on their mobile
devices. There are three different types of ASR
prototypes that are being reviewed. Also the DTMF
prototype in which users enter the commands
through their telephone keyboards is compared with
the ASR prototypes. The important findings of their
study are highlighted in this paper. Also some
limitations are considered and its effect is
discussed.
KEY WORDS:
Interactive Voice Response (IVR), Automatic
speech recognition (ASR) and Dual tone multi
frequency (DTMF).
INTRODUCTION:
Pralhad Surve Prashant Vishe
T.E.Comps T.E.Comps
S.P.I.T,Andheri S.P.I.T.Andheri
survepralhad@gmail.com pvishe@gmail.com
ASR in technical terms can be defined as a process
by which spoken words are converted to text. In
ASR, that is voice enabled IVRS, caller will have to
choose his required menu option by speaking out a
word or a phrase as if he is talking to a human. IVR
is supposed to recognize his speech and provide
him required information as per his choice. Earlier
only digit speech recognisers were used. Now this
limitation is overcome and we have variety of IVR
interaction designs, and a greater variety of
telephone applications.
DTMF is the signal to the phone company that you
generate when you press an ordinary telephone's
touch keys. With DTMF, each key you press on
your phone generates two tones of specific
frequencies. So that a voice can't imitate the tones,
one tone is generated from a high-frequency group
of tones and the other from a low frequency group.
Prior to the now-common speech recognition
software and development toolkits, most telephone-
based interfaces relied on touch-tone input using
the telephone keypad, or Dual Tone Multiple
Frequency (DTMF). Voice processing systems,
which first appeared in the 1980s, used DTMF for
input and recorded human speech as output. The
fixed set of 12 keys (10 digits as well as the # and *
keys) on the keypad lends itself to the construction
of applications that present the caller with a list of
options (e.g., “Press 1 for Sales”), commonly
referred to as menus.
The paper discusses the design of the ASR
services. Interface design is most important with
regards to distinguishing between prompt and
recovery systems.
The system prompt gathers information or
responses from the system users. The current ASR
systems have certain limiting guidelines to be
followed such as, speaking at appointed moment
when the ASR was ready to record or accept the
input (generally "after the beep") and Have to avoid
extraneous vocal sounds, such as throat clearing,
and superfluous speech such as "um-seven".
Prompts have to easy and understandable as the
belief nowadays is that if a user is facing difficulty in
completing a transaction with an automated system
then the system is at fault and not the user. Instead
of training users which seemed like forceful or
unpleasant way of treating the users, these days
powerful error recovery systems have been formed.
Paper also discusses about the evaluation methods.
There are two methods, first is the performance
evaluation which is the conventional testing method
used by the vendors and the second is the more
realistic usability evaluation method. There are
many speech recognition systems available in the
market place. The manufacturers generally carry
out the performance evaluation on their product. It is
given about 98-99% accurate but the condition
under which the testing was carried out is not
mentioned. These conditions play a very significant
role in commenting about the efficiency of the
speech recognition system. As a result we have
usability testing. As performance evaluation has
been crucial for the development of speech
technology systems, in the same way usability
evaluation will be crucial for the development of
services incorporating such systems. Usability
evaluation is concerned with the evaluation of the
design of the application rather than the ability of a
system to perform within that design.
The experiment conducted by the authors use three
prototypes of the ASR. They are:
1. VOICE-1 allows users to interact with the
service in a digit-only scenario, following a
menu prompted by the system
2. VOICE-2 allows users to say command
words instead of digits, following the same
menu prompted by the system.
3. VOICE-3 allows users to say more natural
commands through more natural prompt
Furthermore, a DTMF prototype that allows users to
enter commands only through the telephone
keyboard have been developed, with the purpose of
making a comparison between touch tone and
speech recognition technologies for telephony
services. This is the same as VOICE-1 differing only
in the fact that users use the telephone keyboard to
enter the digit-commands instead of saying them.
A general overview of advantages and
disadvantages of ASR and DTMF are stated as
follows:
ASR
Advantages:
1. It is much easier just to speak out the menu
since mobile phones are small to press
buttons. You need to listen to the menu,
bring it in front to look the screen, press the
button, and then again put the phone in
your ear.
2. We can have unlimited and user friendly
menu options. Further we can browse menu
easily and jump from one menu to another
conveniently.
3. Caller can choose a menu option quickly
without having to listen to boring long list of
menus. IVR can become truly interactive
with its caller using same human voice
4. Using an IVR was always a problem as one
he has to remove it from his ear to look at
the keypad and press a key. This can be
again quite irritating for a voice portal user.
Implementation of ASR completely removes
this problem.
Disadvantages:
1. Accuracy of recognition can be thought as
converting words spoken by a user
accurately to its corresponding text. As per
information available (by Googling and
visiting ASR engine provider websites),
almost all ASR engine has high accuracy
for detecting two words, YES and NO. Apart
from them, other words like numbers, date
of births etc. has lower accuracy in
recognising. For example, accuracy of
recognition of numbers 1, 2, 3 etc vary from
87 to 91% depending on ASR engine.
Accuracy of recognising other natural
English language is fare much worse! As
per report, recognising departments in
company is maximum 85%!
Though accuracy can be improved by
training of ASR, but it is not practical at all
and does not serve a voice portal.
2. Using ASR in a big country where people
speak different languages or same
 language with different accents, speech
recognition accuracy is bound to fare worse.
3. People may design IVR very intelligently to
confirm for any doubtful word recognition by
YES or NO, but it is still irritating for many
people and it slows down the time for
fetching information. In a voice portal where
caller pays by minute of usage, it may not
work in the interest of voice portal users.
4. ASR does not work successfully in every
condition. It may not work in noisy
conditions, weak signal, and disturbance on
lines.
5. It might irritate many people for making
them to talk to a machine. And then it is not
easy to talk some phrases or words like a
conversation.
DTMF:
Advantages:
1. Ease of use in case of landline and while
people using a hands-free for mobile.
2. Very reliable as the DTMF signals does not
get distorted even while speaking
simultaneously.
3. Easy to develop and commission. Faster
response by IVRS
Disadvantages:
1. Finding keys may be difficult as per menu
prompts, especially for new mobile phones
with touch pad only keys.
2. Using a DTMF on a mobile device, one he
has to remove it from his ear to look at the
keypad and press a key. This can be again
quite irritating for a voice portal user.
3. Many people do not understand pulse
dialling, tone dialling, hash buttons, star
button etc.
COMPARISON AND STUDY:
The experiment conducted by Delogu, Di Carlo,
Rotundi, & Sartori, 1998 had some crucial findings.
(a) Speech was preferred to DTMF by a majority of
users; (b) the error recovery procedure used in
VOICE-3 helped the system in solving most of the
problematic situations encountered during the
interaction. (c) speech was judged as being more
satisfying, more entertaining, and easier to use than
DTMF; and (d) user preference for a particular
modality was better predicted by user performance
in nonlinear tasks rather than linear ones.
As an input mechanism, DTMF has the advantage
of being both instantaneous and 100% accurate.
This is in contrast to speech recognition, which is
neither. Processing of speech can cause delays for
a caller, and the variability of spoken inputs will
prevent speech from being 100% accurate in the
foreseeable future. However, given the limitation of
using only 12 keys for input, many users of IVR
systems often feel frustration and hang up. Given
the trade-off between (a) highly accurate yet
severely constrained DTMF input and (b) highly
unconstrained yet often misrecognized speech
input, it is important to determine which input
modality users of a telephone-based information
system would prefer.
Delogu et al. (1998) found no difference in terms of
task completion time and the number of turns per
task when comparing DTMF input for an IVR
system with three different types of speech input.
The three types they experimented with were simple
digit recognition, simple command recognition, and
a full-sentence recognition system. The digit system
only recognized single digits one through nine,
whereas the simple command system recognized a
short list of words such as next, skip, yes, or no.
The full-sentence recognition system recognized
relatively complete sentences, such as “Please skip
to next,” or “Yes, I do.” The analysis of user
attitudes toward the different systems indicated that
users preferred the full-sentence recognition system
to the DTMF input system. The DTMF input system,
however, was preferred to both the simple digit and
the simple command recognition systems.
Users preferred connected word (CW) speech input
(in a CW-based system, users say a string of words
after a system prompt, without any pause required
between the words) and DTMF input systems to an
isolated word (IW) speech input system (in an IW
system, users say only a single word after a
prompt). In addition, they reported an interaction
effect between cognitive abilities (spatial and verbal
abilities) of users and their attitudes toward the
different modalities tested. Users with high spatial
abilities significantly preferred DTMF to CW and IW
input, probably due to the positive effect of high
spatial ability on mental mapping of the DTMF
options. Of interest, users with high verbal abilities
also preferred DTMF to CW and IW input. The
positive effect of verbal skills on the DTMF
preference remains unclear.
Based on this study, the following conclusions can
be drawn. First, in terms of user preference, DTMF
systems are preferred to IW or simple digit
recognition- based voice systems. When compared
to full-sentence or CW recognition systems,
previous studies show inconsistent results—DTMF
systems were rated either better or worse. Second,
in terms of task or system performance, no general
difference has been found between DTMF and
voice systems. For high spatial cognitive users, a
voice system may be better than a DTMF system in
terms of the speed of task performance. Finally,
task or system performance does not necessarily
predict user preference for a particular system.
LIMITATIONS:
The experimental findings of Delogu et al. (1998)
portrayed the foundation of user needs and
requirements. However it had certain drawbacks.
The voice systems used by them did not involve any
true speech recognition technology. They used the
Wizard of Oz methodology to simulate speech
recognition and thus compensated for any
technological barriers. In a study with the Wizard of
Oz methodology, participants either naively believe
or are asked to pretend they are interacting with a
machine, even though in reality it is a human who
interacts with the participants. Therefore, the
applicability of the findings from the study just
mentioned to the design and evaluation of an actual
NL system is more limited. In addition, there was no
statistical analysis of the attitudinal survey data
reported in their paper. Instead, they only provided
the proportions for three questions:
(a) Which of the three prototypes will be better
accepted in the marketplace,
(b) Which is the most enjoyable system, and
(c) Which system will you prefer to use in a real
service?
The previous studies mentioned have the following
shortcomings. First, there has been no systematic
examination of the interaction between the natures
of task and the modality of interaction. There are
certain types of tasks which may be more easily
accomplished with one modality rather than
another. For example, DTMF interaction may be
preferable for a simple linear task (e.g., hearing
messages in the order they are received) whereas
speech interaction may be preferable for
complicated and nonlinear tasks (e.g., hearing a
message from a particular sender in random order).
Inattention to this possible interaction between task
and modality might be responsible for the
inconsistent findings on user preference for full-
sentence recognition systems over DTMF ones.
The lack of general difference in task performance
between DTMF and voice systems can also be
explained by this inattention. Had previous studies
tested and analyzed separately different types of
tasks (e.g., linear vs. nonlinear), they might have
come to a different conclusion.
Second, most previous studies involving full-
sentence recognition systems relied on the Wizard
of Oz methodology. Consequently, this does not
allow any strong conclusions on user preference for
and performance with an actual NL system. In our
study, a state-of-the-art NL system was used to
increase the generalizability of this study. Finally,
user evaluation has been measured in somewhat
naïve ways. It measured user evaluation of a
system by simply asking one or two direct questions
such as “How do you evaluate this system?” or
“How enjoyable was the system?”
CONCLUSION:
First, with regard to task performance as measured
by system effectiveness (task success rate) and
system efficiency (average task completion time),
different modalities favour different types of tasks.
For simple and linear tasks, the DTMF modality was
more effective and efficient, whereas for
complicated and nonlinear tasks, the speech
modality was better.
It should be noted that a function mapping was
provided in the DTMF condition, which probably
favoured this condition. Nevertheless, speech was
better than DTMF for nonlinear tasks. Third,
participants evaluated the speech interface
significantly more positively than the DTMF
interface. Participants of the experiment evaluated
the speech modality as being more satisfying,
entertaining, and natural to use than the DTMF
modality. The speech system was also evaluated as
being easier to use and more entertaining.
Finally, user performance with a modality in simple
linear tasks does not predict well user preference
for that particular modality. Even though linear tasks
could be completed more quickly with DTMF than
with speech, users were generally more satisfied
with the speech modality and more often chose it as
their final preference. Based on the results of high
effectiveness and efficiency of the speech modality
for complicated and nonlinear tasks, user
preference might be better predicted by user
performance in nonlinear tasks.

Another important finding from this study is that

users preferred speech navigation in spite of the
fairly high recognition error rates experienced with
the speech modality. First, part of the reason might
be the “cool factor” of the speech modality. The
speech modality was sophisticated and friendlier.
Another reason for the preference for the speech
modality might be the trade-off that users make
between the advantages of using a speech based
system (e.g., the ability to control a device in a
hands-free mode) and the disadvantage of dealing
with recognition errors. In the domain of messaging,
the need for hands-free control is clear when
attempting to access messages from a cellular
phone. It is also important to consider the issue of
how entertaining and engaging users find their
interaction with a system—in other words the “fun
factor.”

REFERENCES:

Delogu, C., Di Carlo, A., Rotundi, P., & Sartori, D.

(1998). A comparison between DTMF and ASR IVR
services through objective and subjective evaluation

Delogu, C., Di Carlo, A., Rotundi, P., & Sartori, D.

(1998). Usability Evaluation of IVR systems with
DTMF and ASR

http://deepblue.lib.umich.edu/bitstream/2027.42/174
/2/71952.0001.001.pdf
http://www.ivrsworld.com/ivr/ivr-menu-system/

http://www.ivrsworld.com/general/the-big-
question-voice-enabled-menu-or-dtmf-pressed-
menu

http://www.ivrsworld.com/advanced-ivrs/usability-
guidelines-of-ivr-systems