An Introduction to Neural Networks

Prof. Leslie Smith Centre for Cognitive and Computational Neuroscience Department of Computing and Mathematics University of Stirling. lss@cs.stir.ac.uk last ma or update! "# $cto%er &''(! minor update "" )pril &''* and &" Sept "++&! links updated ,they -ere really out of date. &" Sept "++&/ fi0 to math font ,thanks Sietse 1rou-er. " )pril "++2 3his document is a roughly 43ML5ised version of a talk given at the NS6N meeting in 7din%urgh8 Scotland8 on "* 9e%ruary &''(8 then updated a fe- times in response to comments received. Please email me comments8 %ut remem%er that this -as originally ust the slides from an introductory talk:

Overview:
;hy -ould anyone -ant a <ne-= sort of computer> ;hat is a neural net-ork> Some algorithms and architectures. ;here have they %een applied> ;hat ne- applications are likely> Some useful sources of information. Some comments added Sept "++& N7;! ?uestions and ans-ers arising from this tutorial

Why would anyone want a `new' sort of computer?

;hat are ,everyday. computer systems good at... .....and not so good at> Good at Not so ood at 9ast arithmetic Doing precisely -hat the programmer programs them to do @nteracting -ith noisy data or data from the environment Massive parallelism Massive parallelism 9ault tolerance

)dapting to circumstances

;here can neural net-ork systems help>

-here -e can=t formulate an algorithmic solution. -here -e can get lots of e0amples of the %ehaviour -e re?uire. -here -e need to pick out the structure from e0isting data.

What is a neural network?

Neural Net-orks are a different paradigm for computing! von Neumann machines are %ased on the processingAmemory a%straction of human information processing. neural net-orks are based on the parallel architecture of animal brains. Neural net-orks are a form of multiprocessor computer system8 -ith simple processing elements a high degree of interconnection simple scalar messages adaptive interaction %et-een elements ) %iological neuron may have as many as &+8+++ different inputs8 and may send its output ,the presence or a%sence of a short5duration spike. to many other neurons. Neurons are -ired up in a 25dimensional pattern.

Beal %rains8 ho-ever8 are orders of magnitude more comple0 than any artificial neural net-ork so far considered. 70ample! ) simple single unit adaptive net-ork! 3he net-ork has " inputs8 and one output. )ll are %inary. 3he output is & if ;+ C@+ D ;& C @& D ;% E + + if ;+ C@+ D ;& C @& D ;% FG + ;e -ant it to learn simple $B! output a & if either @+ or @& is &.

Al orithms and Architectures!

"he simple #erceptron: 3he net-ork adapts as follo-s! change the weight by an amount proportional to the difference between the desired output and the actual output. )s an e?uation! H ;i G I C ,D56..@i -here I is the learning rate8 D is the desired output8 and 6 is the actual output. 3his is called the Perceptron Learning Rule8 and goes %ack to the early &'(+=s. ;e e0pose the net to the patterns! I$ I% &esired output + + + + & & & + & & & & ;e train the net-ork on these e0amples. ;eights after each epoch ,e0posure to complete set of patterns.

)t this point ,*. the net-ork has finished learning. Since ,D56.G+ for all patterns8 the -eights cease adapting. Single perceptrons are limited in -hat they can learn! @f -e have t-o inputs8 the decision surface is a line. ... and its e?uation is @& G ,;+A;&..@+ D ,;%A;&. @n general8 they implement a simple hyperplane decision surface 3his restricts the possi%le mappings availa%le. &evelopments from the simple perceptron: 1ack5Propagated Delta Bule Net-orks ,1P. ,sometimes kno-n and multi5layer perceptrons ,MLPs.. and Badial 1asis 9unction Net-orks ,B19. are %oth -ell5kno-n developments of the Delta rule for single layer net-orks ,itself a development of the Perceptron Learning Bule.. 1oth can learn ar%itrary mappings or classifications. 9urther8 the inputs ,and outputs. can have real values
'ack(#ropa ated &elta )ule Networks *'#+

is a development from the simple Delta rule in -hich e0tra hidden layers ,layers additional to the input and output layers8 not connected e0ternally. are added. 3he net-ork topology is constrained to %e feedforward! i.e. loop5free 5 generally connections are allo-ed from the input layer to the first ,and possi%ly only. hidden layer/ from the first hidden layer to the second8...8 and from the last hidden layer to the output layer.
"ypical '# network architecture:

3he hidden layer learns to recode ,or to provide a representation for. the inputs. More than one hidden layer can %e used. 3he architecture is more po-erful than single5layer net-orks! it can %e sho-n that any mapping can %e learned8 given t-o hidden layers ,of units..

3he units are a little more comple0 than those in the original perceptron! their inputAoutput graph is )s a function! 6 G & A ,&D e0p,5k.,J ;in C Kin.. 3he graph sho-s the output for k=0.5, , and 0, as the activation varies from 5&+ to &+.
"rainin '# Networks

3he -eight change rule is a development of the perceptron learning rule. ;eights are changed %y an amount proportional to the error at that unit times the output of the unit feeding into the weight. Bunning the net-ork consists of 9or-ard pass! the outputs are calculated and the error at the output units calculated. 1ack-ard pass! 3he output unit error is used to alter -eights on the output units. 3hen the error at the hidden nodes is calculated ,%y back!propagating the error at the output units through the -eights.8 and the -eights on the hidden nodes altered using these values. 9or each data pair to %e learned a for-ard pass and %ack-ards pass is performed. 3his is repeated over and over again until the error is at a lo- enough level ,or -e give up..
)adial 'asis function Networks

Badial %asis function net-orks are also feedfor-ard8 %ut have only one hidden layer.
"ypical )', architecture:

Like 1P8 B19 nets can learn ar%itrary mappings! the primary difference is in the hidden layer.

B19 hidden layer units have a receptive field -hich has a centre" that is8 a particular input value at -hich they have a ma0imal output.3heir output tails off as the input moves a-ay from this point. Lenerally8 the hidden unit function is a Laussian! Laussians -ith three different standard deviations.
"rainin )', Networks!

B19 net-orks are trained %y deciding on ho- many hidden units there should %e deciding on their centres and the sharpnesses ,standard deviation. of their Laussians training up the output layer. Lenerally8 the centres and SDs are decided on first %y e0amining the vectors in the training data. 3he output layer -eights are then trained using the Delta rule. 1P is the most -idely applied neural net-ork techni?ue. B19s are gaining in popularity. Nets can %e

trained on classification data ,each output represents one class.8 and then used directly as classifiers of ne- data. trained on ,08f,0.. points of an unkno-n function f8 and then used to interpolate.

B19s have the advantage that one can add e0tra units -ith centres near parts of the input -hich are difficult to classify. 1oth 1P and B19s can also %e used for processing time5 varying data! one can consider a window on the data! Net-orks of this form ,finite5impulse response. have %een used in many applications. 3here are also net-orks -hose architectures are specialised for processing time5series.

-nsupervised networks:
Simple Perceptrons8 1P8 and B19 net-orks need a teacher to tell the net-ork -hat the desired output should %e. 3hese are supervised net-orks.

@n an unsupervised net8 the net-ork adapts purely in response to its inputs. Such net-orks can learn to pick out structure in their input. Applications for unsupervised nets clustering data! e0actly one of a small num%er of output units comes on in response to an input. reducing the dimensionality of data! data -ith high dimension ,a large num%er of input units. is compressed into a lo-er dimension ,small num%er of output units.. )lthough learning in these nets can %e slo-8 running the trained net is very fast 5 even on a computer simulation of a neural net. .ohonen clusterin Al orithm: 5 takes a high5dimensional input8 and clusters it8 %ut retaining some topological ordering of the output. )fter training8 an input -ill cause some the output units in some area to %ecome active. Such clustering ,and dimensionality reduction. is very useful as a preprocessing stage8 -hether for further neural net-ork data processing8 or for more traditional techni?ues.

Where are Neural Networks applica/le?

..... or are they ust a solution in search of a pro%lem> Neural net-orks cannot do anything that cannot %e done using traditional computing techni?ues8 '-" they can do some things -hich -ould other-ise %e very difficult. @n particular8 they can form a model from their training data ,or possi%ly input data. alone. 3his is particularly useful -ith sensory data8 or -ith data from a comple0 ,e.g. chemical8 manufacturing8 or commercial. process. 3here may %e an algorithm8 %ut it is not kno-n8 or has too many varia%les. @t is easier to let the net-ork learn from e0amples.
Neural networks are /ein used:

in investment analysis!

to attempt to predict the movement of stocks currencies etc.8 from previous data. 3here8 they are replacing earlier simpler linear models. in signature analysis! as a mechanism for comparing signatures made ,e.g. in a %ank. -ith those stored. 3his is one of the first large5scale applications of neural net-orks in the US)8 and is also one of the first to use a neural net-ork chip. in process control! there are clearly applications to %e made here! most processes cannot %e determined as computa%le algorithms. Ne-castle University Chemical 7ngineering Department is -orking -ith industrial partners ,such as Meneca and 1P. in this area. in monitoring! net-orks have %een used to monitor the state of aircraft engines. 1y monitoring vi%ration levels and sound8 early -arning of engine pro%lems can %e given. 1ritish Bail have also %een testing a similar application monitoring diesel engines. in marketing! net-orks have %een used to improve marketing mailshots. $ne techni?ue is to run a test mailshot8 and look at the pattern of returns from this. 3he idea is to find a predictive mapping from the data kno-n a%out the clients to ho- they have responded. 3his mapping is then used to direct further mailshots.

"o pro/e further:

) rather longer introduction ,-hich is more commercially oriented. is hosted %y StatSoft8 @nc. 3here is also another introduction ,including some history. %y Stergiou and Siganos. 3he Natural Computing )pplications 9orum runs meetings ,-ith attendees from industry8 commerce and academe. on applications of Neural Net-orks. Contact NC)9 through their -e%site8 %y telephone DNN ,+.&22" "N('*'8 or %y fa0 DNN ,+.&22" "NO&"' @nternet addresses! #euro#et -hich -as at Pings College8 London8 -as a 7uropean Net-ork of 70cellence in Neural Net-orks -hich finished in March "++&. 4o--ever8 their -e%site remains a very useful source of information $%%% &omputational $ntelligence 'ociety ,-as @777 Neural Net-orks Society. http!AA---.ieee5cis.orgA pu%lish a num%er of ournals on neural net-orks and related areas. @ -rote si0 lectures on 1ack Propagation rather a long time ago8 in a no- defunct ;P. Still8 they are reada%le...

#ewscomp.ai.neural5nets has an very useful set of fre?uently asked ?uestions ,9)Q=s.8 availa%le as a ;;; document at! ftp!AAftp.sas.comApu%AneuralA9)Q.html 0ourses Quite a fe- organisations run courses! -e used to run a & year Masters course in Neural Computation! unfortunately8 this course in in a%eyance. ;e can even run courses to suit you. ;e are a%out to start up a centre in Computational @ntelligence8 called @NC@37. 1ore 2pecific Information Some further information a%out applications can %e found from the %ook Stimulation @nitiative for 7uropean Neural )pplications ,S@7N).. 9or more information on Neural Net-orks in the Process @ndustries8 try ). 1ulsari=s home page . 3he company 1rainMaker has a nice list of references on applications of its soft-are package that sho-s the %readth of applications areas. 3ournals! 3he %est ournal for application5oriented information is #eural &omputing and (pplications8 Springer5Rerlag. ,address! S-eetapple 4o8 Catteshall Bd.8 Lodalming8 LUO 2DS. 'ooks! 3here=s a lot of %ooks on Neural Computing. See the 9)Q a%ove for a much longer list. 9or a not5too5mathematical introduction8 try 9ausett L.8 )undamentals of #eural #etworks8 Prentice54all8 &''N. @S1N + &2 +N""#+ ' or Lurney P.8 (n $ntroduction to #eural #etworks8 UCL Press8 &''O8 @S1N & *#O"* #+2 N 4aykin S.8 #eural #etworks 8 "nd 7dition8 Prentice 4all8 &'''8 @S1N + &2 "O22#+ & is a more detailed %ook8 -ith e0cellent coverage of the -hole su% ect.

Where are neural networks oin ?

) great deal of research is going on in neural net-orks -orld-ide.

3his ranges from %asic research into ne- and more efficient learning algorithms8 to net-orks -hich can respond to temporally varying patterns ,%oth ongoing at Stirling.8 to techni?ues for implementing neural net-orks directly in silicon. )lready one chip commercially availa%le e0ists8 %ut it does not include adaptation. 7din%urgh University have implemented a neural net-ork chip8 and are -orking on the learning pro%lem. Production of a learning chip -ould allo- the application of this technology to a -hole range of pro%lems -here the price of a PC and soft-are cannot %e ustified. 3here is particular interest in sensory and sensing applications! nets -hich learn to interpret real5-orld sensors and learn a%out their environment. New Application areas: Pen PC=s PC=s -here one can -rite on a ta%let8 and the -riting -ill %e recognised and translated into ,)SC@@. te0t. Speech and Rision recognition systems Not ne-8 %ut Neural Net-orks are %ecoming increasingly part of such systems. 3hey are used as a system component8 in con unction -ith traditional computers. ;hite goods and toys )s Neural Net-ork chips %ecome availa%le8 the possi%ility of simple cheap systems -hich have learned to recognise simple entities ,e.g. -alls looming8 or simple commands like Lo8 or Stop.8 may lead to their incorporation in toys and -ashing machines etc. )lready the Sapanese are using a related technology8 fuTTy logic8 in this -ay. 3here is considera%le interest in the com%ination of fuTTy and neural technologies.

2ome comments *added 2eptem/er 4$$%+

Beading this through8 it is a %it outdated! not that there=s anything incorrect a%ove8 %ut the -orld has moved on. Neural Net-orks should %e seen as part of a larger field sometimes called 'oft &omputing or #atural &omputing. @n the last fe- years8 there has %een a real movement of the discipline in three different directions! Neural net-orks8 statistics8 generative models8 1ayesian inference 3here is a sense in -hich these fields are coalescing. 3he real pro%lem is making conclusions from incomplete8 noisy data8 and all of these fields offer something in this area. Developments in the mathematics underlying these fileds have sho-n that there are real similarities in the techni?ues used. Chris 1ishop=s %ook Neural Net-orks for Pattern Becognition8 $0ford University Press is a good start on this area. Neuromorphic Systems 70isting neural net-ork ,and indeed other soft computing. systems are generally soft-are models for solving static pro%lems on PCs. 1ut -hy not free the concept from the -orkstation> 3he area of neuromorphic systems is concerned -ith real5 time implementations of neurally inspired systems8 generally implemented

directly in silicon8 for sensory and motor tasks. )nother aspect is direct implementation of detailed aspects of neurons in silicon ,see 1iological Neural Net-orks %elo-.. 3he main centres -orld-ide are at the @nstitute for neuroinformatics at Murich8 and at the Center for Neuromorphic Systems 7ngineering at Caltech. 3here are also some useful links at this page ,from a UP 7PSBC Net-ork Pro ect on Silicon and Neuro%iology. 1iological Neural Net-orks 3here is real interest in ho- neural net-ork research and neurophysiology can come together. 3he pattern recognition aspects of )rtificial Neural Net-orks don=t really e0plain too much a%out ho- real %rains actually -ork. 3he field called Computational Neuroscience has taken inspiration from %oth artificial neural net-orks and neurophysiology8 and attempts to put the t-o together. 1ack to top of page 1ack to Leslie Smith=s home page
Last updated" *onday, +,!-ul!+00. 0/"+0" 1 2'3 $f you have any difficulties accessing this page, or you have any 4ueries5suggestions arising from this page, please email" Prof Leslie ' 'mith 6lss6nospam7please89cs.stir.ac.uk8