NEURAL NETWORK THEORY

NEURAL DYNAMIC1:
ACTIVATIONHS AND SIGNALS

MAIN POINTS:
NEURONS AS FUNCTIONS()
SIGNAL MONOTONICITY
BIOLOGICAL ACTIVATIONS AND SIGNALS

NEURON FIELDS
NEURONAL DYNAMICAL SYSTEMS
COMMON SIGNAL FUNCTION
PULSE-CODED SIGNAL FUNCTION

NEURONS AS FUNCTION

Figure 1. Neuron Structure Model

Relationship of input-output:

Ii

j 1

ji

x j i

y f I j

NEURONS AS FUNCTION
Common nonlinear transduction description: a sigmoidal or S-shaped curve

Fig.2 s(x) is a bounded monotone-nondecreasing function of x

Signal Function:

1
S ( x)
1 e cx

dS
S'
cS (1 S ) 0
dx

( c 0)

Neurons transduce an unbounded input activation x(t) at time t into a

bounded output signal S(x(t)).

SIGNAL MONOTONICITY
In general, signal functions are monotone nondecreasing S>=0. In
practice this means signal functions have an upper bound or saturation
value.
An important exception: bell-shaped signal function or Gaussian signal
functions

S ( x) e

c0

cx 2

S ' 2cxe cx ,
2

S' x

The sign of the signal-activation derivation s is opposite the sign of the activation
x. We shall assume signal functions are monotone nondecreasing unless stated
otherwise.

SIGNAL MONOTONICITY
Generalized Gaussian signal function define potential or radial basis
function:

1
Si ( x) exp[ 2
2 i
input activation vector:
:
variance:

mean vector:

i 2
(
x

j j) ]
j

x ( x1 , , xn ) R n

i2
i ( 1i , , ni )

Because the function depend on all neuronal activations not just the ith
activation, we shall consider only scalar-input signal functions : S ( x )
i

SIGNAL MONOTONICITY
A property of signal monotonicity: semi-linearity

Comparation:
a. Linear signal functions:
computation and analysis is comparatively easy; do not suppress
noise.
b. Nonlinear signal functions:

Increases a networks computational richness and facilitates noise

suppression; risks computational and analytical intractability;

SIGNAL MONOTONICITY
Signal and activation velocities

the signal velocity:

S =dS/dt
S dS dx S ' x
dx dt

Signal velocities depend explicitly on action velocities. This dependence

will increase the number of unsupervised learning laws.

BIOLOGICAL ACTIVATIONS AND SIGNALS

Introduction to units
Dendrite: input
Axon: output
Synapse: transduce signal
Membrane: potential difference between
inside and outside of neuron
Fig3. Key functional units of a biological neuron

BIOLOGICAL ACTIVATIONS AND SIGNALS

Competitive Neuronal Signal
Signal values are usually binary and bipolar.

Binary signal functions :

Bipolar signal functions :

1
s x
0

x T

1
s x
1

x T
x T

x T

NEURON FIELDS
In general, neural networks contain many fields of neurons. Neurons
within a field are topological.
Denotation:

FX

F X FY FZ

: input field

Fz : output field

Neural system samples the function

s x

m times to generate the associated

pairs x , y , x , y ...... x , y
1
1
2
2
m
m
Classification:

Zeroth-order topological (simplest)

Three-dimensional and volume topological (complex)

NEURONAL DYNAMICAL SYSTEMS

Description:

A system of first-order differential or difference equations that govern the

time evolution of the neuronal activations or membrane potentials
Activation differential equations:

x g FX , FY ,......
y h FX , FY ,......

xi , yi

x x1 , x2 .....

y y1 , y 2 .....

denote the activation time functions of the ith neuron in FX

and jth neuron in

Classification:

FY

Automomous systems: activations are independent of t

Nonautonomous systems: depend on t

NEURONAL DYNAMICAL SYSTEMS

Neuronal State spaces

X t x1 t ,..., x x t R n

Y t y1 t ,... y n t R p

So the state space of the entire neuronal dynamical system is :

n
p

,
F

R
X
Y

Augmentation :

[ FX | FY ] FZ

Fz R n p

Concatenate fields have different computational, metrical or other properties

NEURONAL DYNAMICAL SYSTEMS

Signal state spaces as hypercubes

S ( X (t )) ( S1X ( x1 (t )), , S nX ( xn (t )))

Fig.4 Neural and fuzzy computations conincide.

NEURONAL DYNAMICAL SYSTEMS

Neuronal activations as short-term memory

Short-term memory(STM) : activation

Long-term memory(LTM) : synapse

COMMON SIGNAL FUNCTION

1 Liner Function
S(x) = cx + k

c>0
S
k
o

COMMON SIGNAL FUNCTION

2. Ramp Function
r
if x

S(x)= cx
if |x|<

-r
if x-
r>0, r is a constant.
S

-r

COMMON SIGNAL FUNCTION

3 threshold linear signal function: a special Ramp Function

S ( x) 0
cx

Another form:

if
if

cx 1
cx 0
else

S ( x ) min(1, max(0, cx ))

S' c 0

COMMON SIGNAL FUNCTION

4 logistic signal function:

1
S ( x)

cx
1 e

e
e

c
x
2

c
x
2

c
x
2

Where c>0.

S ' cS (1 S ) 0
So the logistic signal function is monotone increasing.

COMMON SIGNAL FUNCTION

5 threshold signal function:

S ( x k 1 ) S ( x k )
0

if x k 1 T
if x k 1 T
if x k 1 T

Where T is an arbitrary real-valued threshold,and k indicates the

discrete time step.

COMMON SIGNAL FUNCTION

6 hyperbolic-tangent signal function:

S ( x ) tanh( cx )
2

S ' c(1 S ) 0
Another form:

tanh( cx )

e cx e cx
e cx e cx

COMMON SIGNAL FUNCTION

7 threshold exponential signal function:

S ( x ) min(1, e cx )
When

e cx 1

S' ce cx 0
S ' ' c 2 e cx 0
S ( n ) c n e cx 0

COMMON SIGNAL FUNCTION

8 exponential-distribution signal function:

S ( x ) max(0,1 e cx )
When

x0
S' ce cx 0
S ' ' c 2 e cx 0

COMMON SIGNAL FUNCTION

9 the family of ratio-polynomial signal function:
An example

S ( x ) max(0,
For

xn
cx

n 1

S'

cnx n 1
n 2

(c x )

PULSE-CODED SIGNAL FUNCTION

Description:
Pulse trains arriving in a sampling interval seems to be the bearer
of neuronal signal information.
Pulse-coded formulation:

S i (t ) xi ( s )e st ds

S j (t ) y j ( s)e st ds

where

1
xi (t )
0

if a pulse occurs at t
if no pulse at t

xi , yi denote binary pulse functions that summarize the excitation of

membrane potential.

PULSE-CODED SIGNAL FUNCTION

Velocity-difference property of pulse-coded signals

S i (t ) xi (t ) S i (t )

A simple form for the signal velocity:

S j (t ) y j (t ) S j (t )
Current pulse and current signal or expected pulse frequency are
available quantities.
Another computational advantage:
1
0

If xi (t )

if a pulse occurs at t
if no pulse at t

1 s t 0
s t 0

a sustained pulse arrived

no pulse arrived