1 INTRODUCTION 1.

1 PROBLEM DEFINITION
Project aims at recognizing spoken words and represents the words using American standard sign language via a robotic arm and also on the computer using visual basic. The application uses MATLAB to achieve it. Our idea behind this project is to help the dea and the dumb recognize the words spoken more e icientl!.

1.2 FIELD INFORMATION COLLECTED

1.2.1 AVAILABLE METHODS OF RECOGNITION There are man! methods available to recognize the identi ied eatures and classi ! them into categories. The most commonl! used methods are "MM# A$$# A$%&' and M%((. )e have developed M%(( in our project# to recognize the speech. "MM* "idden Markov Model A$$* Arti icial $eural $etwork A$%&'* Adaptive $euro+%uzz! &n erence '!stem M%((* M,LL %re-uenc! (epstral (oe icient

Our irst task was to ind out the wa! or recognizing speech. There are man! methods available or speech recognition. 'peech recognition can be done using "idden Markov Models .sing $eural $etworks /!namic Time )arping /igital 'ignal Processing Mel %re-uenc! (epstrum

 Log 0Likelihood A$%&'

)e inall! choose the method o mel re-uenc! trans ormation. The most widel! used eature in speech recognition is mel re-uenc! coe icients# which are mainl! ocused to model vocal tract while e1tracting eature coe icients rom the speech# mel re-uenc! cestrum coe icients appl! mel scale to power spectrum o speech in order to imitate human hearing mechanism. %ilters are used to emphasize some o the re-uenc! contents like ear does which is also called perceptual weighting. More ilter banks process the spectrum below 2 k"z since the speech signal contains most o its use ul in ormation such as irst ormant in lower re-uencies.

1.3 ABOUT SIGN LANGUAGE

'ign language is the language used b! dea and mute people. &t is a combination o shapes and movements o di erent parts o the bod!. These parts include ace and hands. The area o per ormance o the movements ma! be rom well above the head to the belt level. 'igns are used in a sign language to communicate words and sentences to audience. A gesture in a sign language is a particular movement o the hands with a speci ic shape made out o them. %acial e1pressions also count toward the gesture# at the same time. A posture on the other hand# is a static shape o the hand to indicate a sign. A sign language usuall! provides signs or whole words. &t also provides signs o letters to per orm words that don3t have a corresponding sign in that sign language. 'o# although sentences can be made using the signs or letters# per orming with signs o words is aster. The sign language chosen or this project is the American 'ign Language. AMERICAN STANDARD SIGN LANGUAGE

&t is the most well documented and most widel! used language in the world. American 'ign Language 4A'L5 is a comple1 visual+spatial language that is used b! the /ea communit! in the .nited 'tates and ,nglish+speaking parts o (anada &t is a linguisticall! complete# natural language. &t is the native language o man! /ea men and women# as well as some hearing children born into /ea amilies. A'L shares no

grammatical similarities to ,nglish and should not be considered in an! wa! to be a broken# mimed# or gestural orm o ,nglish. American standard sign language was demonstrated around 2678 in .nited 'tates and man! other countries# to provide education or the people who have problem in speaking and hearing the words 4the de and the dumb5. A+'+'+L enables the dumb and dea to speak with their hands which represent their tongue.

%ig. 2.2 ,nglish cue s!mbols

2. Literal Sur e!
2.1 SURVE" OF SIGN LANGUAGE
9arious s!stems were proposed or the automatic recognition o sign language /on Pearson in his approach :9isual (ommunication '!stems or the /ea ; presented a two wa! communication approach# where he proposed the practicalit! o switched television or both dea +to+hearing and dea to+dea communication. &n his approach# attention is given to the

re-uirements o picture communication s!stems# which enable the dea to communicate over distances using telephone lines. This section discusses some research done on translating other te1t and spoken languages to 'ign language. (o1 et. al# <2= presented a s!stem that translates the ,nglish speech to the British 'ign Language 4B'L5 using a speciall! developed avatar. "owever# the s!stem is constrained or post o ice operations. The s!stem uses a phrase lookup approach due to the highl! constraint in the Macintosh operating s!stem. The authors divided the task into three di erent problems* Automatic speech to te1t conversion 2. Automatic translation o arbitrar! ,nglish te1t into suitable representation o &ndian sign language. >. /ispla! o this representation as a se-uence o sign using computer graphics techni-ues /eveloped s!stem achieved accurac! o identi ication o the signed phrases o ?2@ or complete phrases A 72@ or sign units. "owever# the eedback o dea users and post o ice clerks were ver! encouraging or urther development. A group o >2 researchers at /ePaul .niversit! participated in developing an automated American sign Language '!nthesizer. 'uszczanska. et. <>= developed a s!stem to translate te1ts written in Polish Language into Polish 'ign Language. The! used Avatar as well with a dictionar! o ?88 signs. 'carlatos. ,t.al. introduced a s!stem to translate speech into video clip o the American 'ign Language 4A'L5. The s!stem displa!s the A'L clips along with the written words. The! used a built+in speech recognition engine in the Macintosh operating s!stem. This added a limitation as this engine can onl! recognize words rom a pre+de ined set.

The! plan to e1tend the s!stem to recognize more words and later or phrases. 'an+'egundo# and others# developed a s!stem to translate speech into 'panish 'ign Language. Their s!stem is made up o our modules* speech recognizer# semantic Anal!sis # gesture se-uence generation and gesture pla!ing. %or the speech recognizer# the! used modules developed b! &BM. %or the semantic anal!sis the! used modules developed b! the .niversit! o (olorado. %or gesture se-uence generation the! used the semantic concepts associated to several 'panish 'ign

Language gestures. %or gesture animation the! developed an animated character and a strateg! or reducing the , ort in gesture generation . The strateg! consists o making the s!stem generate automaticall! all agent positions necessar! or the gesture animation Towards the development o automated speech recognition or vocall! disabled people a s!stem called :Bolta! "aath; <B= is developed to recognize :Pakistan 'ign Language;4P'L5 at 'ir '!ed universit! o engineering and Technolog!. The Bolta! "aath project aims to produce sound matching the accent and pronunciation o the people rom the sign s!mbol passed. A wearing /ata Clove or vocall! disabled is designed# to trans orm the signed s!mbols to audible speech signals using gesture recognition. The! use the movements o the hand and ingers with sensors to inter ace with the computer.

2.2 PURPOSE
To represent the American 'tandard 'ign Language this can be understood b! the dea and the dumb. De#i$% &et'()(l($! /esign o the paper is based on the principle o Dspeech recognition3 where the speech is converted to the te1t and passes the command as isolated words to hardware which represent the signs. Fi%)i%$ To provide a so tware package to convert the speech signal# into the sign language.

S(*ial *au#e To bridge the communication and e1pression gap between the normal people who cannot understand the sign language# and the dea and dumb who cannot understand the normal speech.

3. COMPONENT SELECTION
3.1 Mi*r(*(%tr(ller Ba#i*#
6

)hat are microcontrollers and what are the! used orE As all other good things# this power ul component is basicall! ver! simple and is obtained b! uniting tested and high+ -ualit! :ingredients; 4components5 as per ollowing receipt* 2. The simplest computer3s processor is used as a :brain; o the uture s!stem. >. /epending on the taste o the producer# it is added* a bit o memor!# a ew AF/ converters# timers# inputFoutput lines etc. G. &t is all placed in one o standard packages. B. 'imple so tware that will be able to control it all and about which ever!one will be able to learn has been developed.

Mi*r(*(%tr(ller Sele*ti(%
A microcontroller is basicall! selected depending upon 2. Availabilit! o component. >. Availabilit! o development tools G. Present market trend ARM+ ,LPC213-. The LP(>2G7 microcontrollers are based on a G> bit AHMI with real+time emulation and embedded trace support that combines the microcontroller with J2> kB o embedded high speed %lash memor!. /ue to their tin! size and low power consumption# these microcontrollers are ideal or applications where miniaturization is a ke! re-uirement# such as access control and point+o +sale.

3.2 SERIAL COMMUNICATION BASICS

All &BM P( and compatible computers are t!picall! e-uipped with two serial ports and one parallel port. Although these two t!pes o ports are used or communicating with e1ternal devices# the! work in di erent wa!s. A parallel port sends and receives data eight bits at a time over 7 separate wires. This allows data to be trans erred ver! -uickl!K however# the cable re-uired is more bulk! because o the number o individual wires it must contain. Parallel ports are t!picall! used to connect a P( to a printer and are rarel! used or much else. A serial port sends and receives data one bit at a time over one wire. )hile it takes eight times as long to trans er each b!te o data this wa!# onl! a ew wires are re-uired. &n act# two+wa! 4 ull duple15 communications is possible with onl! three separate wires + one to send# one to receive# and a common signal ground wire. $otice that when onl! I data bits are emplo!ed# !ou cannot send A'(&& values greater than 2>I. Likewise# using J bits limits the highest possible value to G2. A ter the data has been transmitted# a stop bit is sent. A stop bit has a value o 2 0 or a mark state 0 and it can be detected correctl! even i the previous data bit also had a value o 2. This is accomplished b! the stop bit3s duration. 'top bits can be 2# 2.J# or > bit periods in length

PINS OF DB/0 CONNECTOR

%ig G.2 /B+6 (onnector Pin Out

Table G.2 /B+6 (onnector Pin Out DB/0 C(%%e*t(r Pi% Out Pi% 1 Si$%al Na&e 1 2 3 3 4 5 + 0 (/ HL/ TL/ /TH C$/ /'H HT' (T' H& Si$%al De#*ri2ti(% (arrier /etect Heceive /ata Transmit /ata /ata Terminal Head! 'ignal Cround F (ommon /ata 'et Head! He-uest To 'end (lear To 'end Hing &ndicator

T6D7 Transmit /ataK the data sent rom the /ata Terminal and received b! the /ata 'et. R6D7 Heceive /ataK the data sent rom the /ata 'et and received b! the /ata Terminal. DTR7 /ata Terminal Head!K .sed b! the /ata Terminal to signal to the /ata 'et that it is read! or operation# active high. DSR7 /ata 'et Head!K .sed b! the /ata 'et to signal to the /ata Terminal that it is read! or operation and read! to receive data# active high. RTS7 He-uest To 'endK .sed b! the /ata Terminal to signal the /ata 'et that it ma! begin sending data. The /ata 'et will not send out data without this signal# active high. CTS7 (lear To 'endK .sed b! the /ata 'et to signal the /ata Terminal that it ma! begin sending data. The /ata Terminal will not send out data without this signal# active high.

CD7 (arrier /etectK .sed b! the /ata 'et to indicate to the /ata Terminal that the /ata set has detected a carrier 4o another device5. RI7 Hing &ndicatorK .sed b! the /ata 'et to indicate to the /ata Terminal that a ringing condition has been detected. GND7 CroundK The common return or all signals on the inter ace.

3.3 CONSIDERATIONS FOR LCD

Li8ui) Cr!#tal Di#2la!# ,LCD. These components are :specialized; or being used with the microcontrollers# which means that the! cannot be activated b! standard &( circuits. The! are used or writing di erent messages on a miniature L(/.

%ig G.> L(/ A model described here is or its low price and great possibilities most re-uentl! used in practice. &t is based on the "/BBI78 microcontroller 4"itachi5 and can displa! messages in two lines with 2? characters each. &t displa!s all letters o alphabet# Creek letters# punctuation marks# mathematical s!mbols etc. &n addition# it is possible to displa! s!mbols that user makes up on its own. Automatic shi ting message on displa! 4shi t le t and right5# appearance o the pointer# backlight etc. are considered as use ul characteristics.

10

3.3 Ser ( M(t(r

'ervo motor consists o several main parts# the motor and gearbo1# a position sensor# an error ampli ier and motor driver and a circuit to decode the re-uested position. %igure 2 contains a block diagram o a t!pical servo motor unit. The radio control receiver s!stem 4or other controller5 generates a pulse o var!ing length appro1imatel! ever! >8 milliseconds. The pulse is normall! between 2 and > milliseconds long. The length o the pulse is used b! the servo to determine the position it should rotate to. The current rotational position o the servo motor output sha t is read b! a sensor. This is normall! a potentiometer 4variable resistor5 which produces a voltage that is related to the absolute angle o the output sha t. The position sensor then eeds its current value into the ,rror Ampli ier which compares the current position with the commanded position rom the pulse width to voltage converter. Man! microcontrollers are e-uipped with P)M generators and most people initiall! consider using these to generate the control signals. The problem is that we need a relativel! accurate short pulse then a long dela! and generall! !ou onl! have one P)M generator share between several servos which would re-uire switching components outside the microcontroller. The P)M generator is designed to generate an accurate pulse between 8@ and 288@ dut! c!cle# but we need something in the order o J@ to 28@ dut! c!cle 42msF>8ms to >msF>8ms5.

11

3.1 Bl(*9 Dia$ra&

%ig. B.2 Ceneral block diagram o the project

12

3.2 DESCRIPTION OF BLOC: DIAGRAM

Block diagram o the overall s!stem will be implemented. The s!stem will be operating in close to real time and will take the speech input rom the microphone and will convert it to s!nthesized speech or inger spelling. 'peech recognition will be implemented or the considered languages. Language models will be used to solve ambiguities. %inger spelling s!nthesis will be implemented. Language model in matlab is used to s!nthesis speech in to te1t. And given to arm controller which will have L(/ to veri ! words spell and processed b! processor. Hobotic arm is connected to arm to represent the vocal language in to sign language. The robotic arm uses servo motors or their movement. 'peech recognition s!stem based on the speech reading and the samples passed to the processing unit. The processing s!stem consists o a speech recognition unit with s!mbol generator# which determines the speech signal and produces an e-uivalent coded s!mbol or their cognized speech signal.

3.3 SPEECH RECOGNITION B" MFCC

3.3.1 ;(r9i%$ &n P(# using speech recognition algorithm in MATLAB speech will get recognized taking input rom microphone. Then that recognized speech will get trans er seriall! to the AHMI controller. To the ArmI controller# servo motors are inter aced so according to letter or word the movement o servo will be done to move hand assembl! according sign languages. %ollowing steps are taken to recognition o speech* 'peech is one o the oldest and most natural means o in ormation e1change between human beings. )e humans speak and listen to each other in humanMhuman inter ace. %or centuries people have tried to develop machines that can understand speech as humans do so naturall!.

13

%or doing so we have used the method o M%(( coe icients. )e have used M%(( because it resembles the human ear in man! wa!s.

The output o the human ear is linear above 2 k"z and so is the output o M%(( and the output is logarithmic above 2 k"z or the human ear as well as M%((. 'o it resembles the human auditor! eatures.

3.3.2 Mel/Fre8ue%*! Tra%#<(r&ati(% According to ps!choph!sical scientists# "uman hearing perception o re-uenc! contents is not linear. &t is linear up+to 2888 "z and logarithmic over 2888 "z. This logic can be implemented b! using this ormula# Mel+ re-uenc! M>J6JNlog 42Olinear re-uenc!FI885 Mel re-uenc! cepstral coe icients ,ach region on the basilar membrane acts as a ilter bank. There are more receptors or re-uencies rom 8+2Phz and the number decrease rapidl! thereon. Mel ilter banks are used to simulate this

14

SPEECH RECOGNITION B" MFCC

There are two phase involved in speech recognition process* 25 THA&$&$C >5 H,(OC$&T&O$ The overview o the M%(( process* %,AT.H, ,LHTA(T&O$

15

%ig B.> Block diagram speech recognition b! m cc

4. FUTURE SCOPE
B! American 'tandard 'ign Language with speech recognition #it is possible to speak directl! with a robot without an! ph!sical engagement to communicate with the dumb and dea easil!. &t is easil! possible to make a so tware package to automaticall! install in video and movie services with the help o set top bo1 and record the speech words and at the time o testing 4pla!ing the movie5 represent the sign language b! animated hand at the corner o screen.

16

5. REFERENCES
<2= (o1 et. al.#Automated Speech Recognition Approach to Continuous cue-Symbols Generation, &nternational Qournal o Power (ontrol 'ignal and (omputation 4&QP('(5 9ol. > $o. 2# March 266>. <>= '. %urui#Speaker independent isolated word recognition using dynamic features of speech spectrum# &,,, Transactions on Acoustic# 'peech# 'ignal Processing# 9ol. GB# $o. 2# pp. J>+J6# %ebruar! 267?. <G= Mahdi 'haneh# and Azizollah Taheri# oice Command Recognition System !ased on "#CC and $ Algorithms# )orld Academ! o 'cience# ,ngineering and Technolog!. <B= Andrew 'loss# :AHM user guide;# Person Publication. <J= Haj Pamal# :,mbedded s!stems; McCraw+"ill# 'econd ,dition. <?= http*FFen.www.)ikipedia.orgF wikiFAmericanR'ignRLanguage. <I= http*FFwww.alldatasheet.comFview.jspE'earchwordMLP(>2G7.

17