GSTF Journal of BioSciences (JBio) Vol.3 No.
1, August 2014
Fish School System Identification and Control
Based on Artificial Neural Network
Ammar Ibrahem Majeed
Electrical Engineering Department
University of Basrah
Basrah, Iraq
e-mail: ammar_19722@yahoo.com
Abstract—The purpose of this paper is to introduce Artificial
Neural Network (ANN) based system identification for a
nonlinear dynamics structure of fish school for trajectory
tracking problem. A discrete time dynamic model has been
derived, and the neural network (NN) training data has been
collected experimentally using a Simulink model within
MATLAB program designed for this purpose. This work also
seeks to design a user friendly program to automatically
monitor and track the trajectory of the live fish in an aquarium
using two webcams to produce a three dimensions (3D) view of
the acquired trajectory. MATLAB program was used to
develop a graphical user interface (GUI) which allows the
monitoring user to input experiment specifications such as
desired tracking time, tracking region and video source (real
time directly from webcams vs. cached from the pre-saved
files) in addition displaying the acquired videos from two
webcams, position of the tracked fish (trajectory), and fish
velocity. A predictive control approach based on NN is
proposed to provide the control signal(s) that could be used to
control the robotic fish to achieve trajectory tracking task in
an attempt to design and build autonomous robotic fishes in
future that are able to reactive to the environment and
navigate toward the desired location underwater.
Keywords—Trajectory; Tracking; Fish Identification; Artificial
Neural Networks; Simulink & MATLAB.
I. INTRODUCTION
In nature, fish has astonishing swimming ability after
thousands years evolution. It is well known that the tuna
swims with high speed and high efficiency, the pike
accelerates in a flash and the eel could swim skilfully into a
narrow hole. Such astonishing swimming ability inspires us
to improve the performance of aquatic man-made robotic
systems, namely Robotic Fish. Instead of the conventional
rotary propeller used in ship or underwater vehicles, the
undulation movement provides the main energy of a robotic
fish. The observation on a real fish shows that this kind of
propulsion is more noiseless, effective, and manoeuvrable
than the propeller-based propulsion [1].
II. MODELLING A ROBOTIC FISH
The fish model was built by using MATLAB in a 3D
graph as shown in figure (1).
Fig. 1. MATLAB 3D robotic fish model.
DOI: 10.5176/2251-3140_3.1.50
42
Abduladhem Abdulkareem Ali, SMIEEE
Computer Engineering Department
University of Basrah
Basrah, Iraq
e-mail: abduladeeml@yahoo.com
The animation program was also written using MATLAB
programming language and OpenGL is adopted for the
animation display. As shown in Figure (1), the fish trunk
and fish fins are designed simultaneously. The former
consists of seven parts: the nose, the neck plane, the first
joint plane (J1), the second joint plane (J2), the third joint
plane (J3), the forth joint plane (J4) and the tail. Each part is
defined by a group of six traverse points (A, B, C, D, E, F)
which are used to compute the control points of bezier-
spline in the OpenGL render processing. These points are
arranged into two groups: (ABCD) and (AFED), to build the
right-half and the left-half fish body respectively [1]. The
values of these traverse points are proportional to the real
robotic fish. It is convenient to change the size of the robotic
fish model by changing the traverse point's positions by the
slide bars at the right top corner of the MATLAB graphic
user interface (GUI) menu. In each joint plane, an axis
through points A and D acts as the axis of a virtual servo
motor which represents the servo motor in the real robotic
fish. The model could animate fish-like movement when the
virtual servo motors turn by each axis in a special way. The
robotic fish fins include dorsal fin, left pectoral fin and right
pectoral fin. They are designed for future balance control
and up-down motion control.
III. NEURAL NETWORK BASED DYNAMIC MODEL
A major proportion of any control project is spent trying
to form an adequate model of the plant [2]. This is where it
is claimed that neural network have a great advantage over
classical techniques because they can be trained with
observed data from the real plant (or an existing model of
the plant), to reproduce the characteristics of the plant, as a
black box model. The neural networks are used as nonlinear,
multi-variable function approximation tools.
The 'training' involves the optimization of the weights in
the network until the knowledge distributed among them is
sufficient to provide a black-box model of the plant. There
are different methods of adjusting the weights (learning
algorithms), and different neural architectures. For the
purposes of this paper we will use the neural networks for
function approximation. As shown in figure (2), we have
some unknown function that we wish to approximate. We
want to adjust the parameters of the network so that it will
produce the same response as the unknown function, if the
same input is applied to both systems [3].
Recent results show that neural-network techniques seem
to be very effective to identify a wide class of complex
nonlinear systems when we have no complete model
information, or even when we consider the controlled plant
as a black box. A possible way to describe a system is in the
form of the following discrete equation (u is input, y is
output).
© 2014 GSTF
 GSTF Journal of BioSciences (JBio) Vol.3 No.1, August 2014

next frame from the two webcams, and so on, in order to

reconstruct the 3D trajectory for each group member.

For our applications, the unknown function may

correspond to a system we are trying to control, in which
case the neural network will be the identified plant model.
The unknown function could also represent the inverse of a
system we are trying to control, in which case the neural
network can be used to implement the controller.

Fig. 3. A schematic diagram of the experimental setup.

V. COLLECTING THE TRAINING DATA AND

MATLAB/SIMULINK PROGRAM
The Simulink model shown in figure (4) has been
constructed in order to take the image and performed the
following functions; the video data from the two webcams
were each split into RGB (Red, Green, and Blue)
Fig. 2. Neural Network used for Function Approximation. components. Two of the components (R and B) were used
(the combination that best resulted in bringing out the fish
The neuro-control shows that if a disturbance occurs in from the background) and added together. This output was
the system, the neural network learns to counteract this multiplied by a constant to saturate the pixels slightly for the
disturbance. Finally the knowledge learned in identification upcoming functions. Next, a threshold value was applied
and control was applied to the real time fish model. An that turned the image binary. A binary image is strictly
online adaptive neural network was developed to model the black and white, pixels having values 0 (white) or 255
real time system. (black). The threshold value was an environmentally
The construction of a dynamic model with neural sensitive constant that determined the limit for which pixels
networks is a more difficult task than that of a static model. turned black and which turned white. To smooth the edges
To ensure that past inputs are considered, the problem can around the silhouettes a dilate operator ‘enlarged’ the pixels,
be transformed from a temporal one to spatial one by having them ‘bleed’ into each other, overlapping data areas
supplying the past values of inputs and outputs as new input and smoothing the contours. This sequence of filtering was
dimensions. An alternative to this is the use of recurrent applied to highlight the fish with black and white
networks, which contain internal feedback connections. Two background disturbances, see figure (5).
feed forward network types, the Multi-Layer Perception
(MLP) or Radial Basis-Function Networks (RBF), may be
used to model the system. The use of neural networks to
describe a non-linear dynamic system is based on the
assumption that the system can be represented by a non-
linear auto regressive moving average model with
exogenous inputs (NARMAX) over the whole operation
envelope. The validity o f this assumption is discussed by
Leontaritis and Billings [4]. The training data for the neural
network was collected experimentally using a Simulink
model within MATLAB program.
IV. EXPERIMENTAL SETUP
In order to obtain dynamic information on fish school
motion, individuals must be tracked in time in 3D bases
using either one or two webcams [4]. In this paper we
choose the two webcams method as shown in figure (3); that
is, the video from these webcams must be splitted into
Fig. 4. Black and white filtering Simulink functions.
frames and then processed frame by frame in order to get
the 3D fish trajectory. An individual in one frame from the
two webcams must be linked to the same individual in the

© 2014 GSTF
43
 GSTF Journal of BioSciences (JBio) Vol.3 No.1, August 2014
Fig. 5. Results of black and white filtering for two webcams using Simulink
model.
The fish was highlighted in a black and white video
stream with the above filtering scheme. To single out the
fish with color background, more filtering was required.
This signal was complimented, taking a binary image and
inverting the value of the pixels. White pixels were turned
black, and black pixels were turned white. This image was
multiplied by a constant to prevent over saturation of a
contrasty image. The next several operations were the
results of plug-and-play experimentation that again
emphasized the fish. A series of logical operators were
applied to the image, XNOR, NAND, and OR functions.
These logical operators were used in series and parallel with
each other, resulting in an image that would function with
the latter parts of the Simulink functions. The Simulink
model for this color filtering scheme is shown in figure (6).
Fig. 6. Color filtering subsystem.
After the filtering created a video emphasizing our target
fish, the model took this video feed and determined the
locations of the fish. Singling out the fish was the task of a
function called “blob analysis” shown within the Simulink
model in figure (4). This function took the image and looked
for pixels in a similar region with similar values. These
“blobs” of pixels were further defined with a user inputted
value of maximum and minimum blob sizes. For our
program, the minimum blob size was approximately the
viewable pixel area of a fish swimming directly to or away
from the camera. The maximum blob size was the
approximate pixel area of a fish seen from its side at a near
distance to camera. With these parameters set, the centroid
function selected reasonable objects (blobs) from the video
feed and assigned central coordinates (centroids) to these
objects. These centroid values are the position of the fish.
Figure (7) shows the results of fish blob analysis and
centroid function.
44
Fig. 7. The results of fish blob analysis and centroid function and then fish
school tracking.
The superimposed yellow centroids shown in figure (7)
would track the fish individuals during motion and gives the
exact (x, y and z) coordinates for the 3D trajectory tracking
of each individual. These coordinates are then stored in a
matrices for using in neural network training.
VI. GUI PROGRAM DESIGN
Simulink’s video and image processing toolbox gives
users an array of functions to modify video feeds. Other
functions outside of the video and image processing toolbox
were also applied to the video feed. However, the command
line of MATLAB held a wider array of functions to
implement on the video feed. Similar models achieving
similar tasks have been created in the command line. As a
work in continual progress, in order to track objects the
designed program using MATLAB’s commands line
performed a series of functions. Users were initially poised
with a Graphical User Interface (GUI) that enables them to
select the required experiment requirements within three
display menus, named (Tracking, Data Graphing, and
configuration), see figure (8). Within the Tracking menu we
can choose the region of interest to record within the fish
tank, the video source, sampling rate, and the tracking time.
And within the Data Graphing menu we can draw the 3-
dimensional travel paths for selected objects which
generated for analysis of travel path characteristics and fish
behavior. The graph of velocity and acceleration of the
tracked objects also could be chosen for drawing. In the
configuration menu, the parameters such as tracked fish
size, camera calibration, background dilation, secondary
dilation could be configured in order to take a video feed
from a webcam for the region of interest in the fish tank and
generate travel path data and also to isolate the desired
objects to track. Figure (9) shows the fish school trajectory
tracking drawing for a fish tank containing four species.
Each fish was given a unique color for its trajectory path for
identification purposes.
© 2014 GSTF
 GSTF Journal of BioSciences (JBio) Vol.3 No.1, August 2014

Fig. 9. Fish school trajectory tracking result.

VII. NEURAL NETWORK PREDICTIVE CONTROL

The neural network predictive controller uses a neural
network model of a nonlinear plant to predict future plant
performance [6]. The controller then calculates the control
input that will optimize plant performance over a specified
future time horizon. The first step in model predictive
control is to determine the neural network plant model
(system identification). Next, the plant model is used by the
controller to predict future performance.
The following section describes the system identification
process. This is followed by a description of the
optimization process. Finally, it discusses how to use the
model predictive controller block that has been implemented
in Simulink.
VIII. SYSTEM IDENTIFICATION
The first stage of model predictive control is to train a
neural network to represent the forward dynamics of the
plant. The prediction error between the plant output and the
neural network output is used as the neural network training
signal. The process is represented by the block diagram
shown in figure (10).

Fig. 10. System identification process.

The neural network plant model uses previous inputs and

previous plant outputs to predict future values of the plant
output. The structure of the neural network plant model is
given in figure (11).

Fig. 8. GUI for fish school tracking software.

Fig. 11. Neural network plant model.

© 2014 GSTF
45
 GSTF Journal of BioSciences (JBio) Vol.3 No.1, August 2014
This network can be trained offline in batch mode, using
data collected from the operation of the plant.
IX. PREDICTIVE CONTROL
The model predictive control method is based on the
receding horizon technique [6]. The neural network model
predicts the plant response over a specified time horizon.
The predictions are used by a numerical optimization
program to determine the control signal that minimizes the
following performance criterion over the specified horizon.
where N1, N2 and Nu define the horizons over which the
tracking error and the control increments are evaluated. The
u' variable is the tentative control signal, yr is the desired
response and ym is the network model response. The ρ value
determines the contribution that the sum of the squares of
the control increments has on the performance index.
The block diagram shown in figure (12), illustrates the
model predictive control process. The controller consists of
the neural network plant model and the optimization block.
The optimization block determines the values of u' that
minimize J, and then the optimal u is input to the plant. The
controller block has been implemented in Simulink, as
described in the following section.
Fig. 12. Model predictive control process.
X. USING THE NN PREDICTIVE CONTROLLER
This section demonstrates how the NN Predictive
Controller block of MATLAB program is used. The
program begins by importing the training positioning data
collected before by tracking program to train the Simulink
NN plant model (System Identification). The potential
training data is then drawn and is shown in figure (13).
Fig. 13. Importing and drawing the plant model training data.
46
The training proceeds according to the selected training
algorithm (trainlm in this case). This is a straightforward
application of batch training. After the training is complete,
the response of the resulting plant model is displayed, as
shown in figure (14). This figure also shows plots for
difference or error between plant output and neural network
model output and neural network plant model output (one
step ahead prediction).
Fig. 14. The response of the resulting plant model
The result of simulating the closed loop system is shown
in figure (15). This graph shows the reference signal and the
system response for the final predictive controller (using the
neural network trained with the data shown in figure (13)
and controller parameters set to N2 = 7, Nu= 2, = 0.05).
Steady state errors were small, and the transient
performance was adequate in all tested regions. It found that
the stability was strongly influenced by the selection of .
As  decreased, the control signal tends to change more
abruptly, generating a noisy plant output. As with linear
predictive control, when  increased too much, the control
action is excessively smooth and the response is slow, this is
clear in some places in graph of figure (15).
Fig. 15. System response and reference signal using the predictive
controller
XI. CONCLUSION
In this paper, a moving objects detection algorithm is
proposed to process fish school videos. The AVI video splits
into a sequence of frames (images) and each frame is
processed to extract data like position, area and number of
individuals of all frames and these data concatenated in one
matrix. Then, this data in matrix is used to compute the
velocity and to detect a path of each object. The collected
data is also used to make system identification of fish
school. A number of difficulties were solved for processed
video of fish individuals. All experiments are done inside
laboratory then experiments can be repeated several times to
© 2014 GSTF
 GSTF Journal of BioSciences (JBio) Vol.3 No.1, August 2014

get good videos suitable for image processing. The image Basrah. His research interests are in biomimetics and control of multiple
swimming robots.
processing algorithm is very sensitive to the environment of
the aquarium under monitoring and the illumination of the Abduladhem A. Ali received his M.Sc. and Ph.D.
aquarium, in addition to that the fish body colour adds degrees from the Department of Electrical
another problem in tracking algorithm therefore we use a Engineering, University of Basrah, Iraq, in 1983 and
1996. He worked as Assistant Lecturer, Lecturer, and
white background to overcome this problem. White Assistant Professor in the same Department from
background, permit mostly automatic detection of 1984, 1987, and 1991, respectively, and then as
individuals in images, and the videos are acquired indoor Assistant Professor and Professor in the Department
without disturbing nature behavior like wind, waves, etc. of Computer Engineering from 1997 and 2004,
respectively. He has worked as a consultant to many
pieces Images reflection from aquarium surfaces was also industrial firms to design industrial control systems. He has published more
treated to enhance the accuracy of the acquired data. than 70 papers, has one patent, and has supervised many M.Sc. and Ph.D.
From these results, many information from the fish school dissertations. He holds the Editor chair for the Iraqi Journal for Electrical
videos can be extracted, such as position, speed, and and Electronic Engineering and is a member of the editorial board for many
acceleration in addition to the main goal of this paper which journals. He was Chairman of the first IEEE International conference on
Energy, Power and Control (EPC-IQ01). His fields of interest are robotics,
is the path trajectory tracking which is useful for several industrial control and intelligent systems. He was the Director of Avicenna
applications. The parameters extracted from the available E-learning center at the University of Basrah, Iraq.
schools were used to train multi-layered feed-forward
artificial neural networks. Various applied networks easily
generated associations between school descriptors and
species identity. Three-dimensional travel paths for selected
objects were generated for analysis of travel path
characteristics and fish behavior. Velocity and acceleration
data was an optional requirement. System identification of a
fish school system was performed using the training data
collected by the MATLAB tracking program. The
experiments requirements were setup through the GUI of the
MATLAB program. Finally the NN system model of a fish
school was used for constructing the predictive controller.
The controller was then used in the future to control the
servo motors of the robotic fish(s) to track the trajectory of
the live fish(s), which is the goal of our next paper.
REFERENCES
[1] J. Liu and H. Hu, “Building a 3d simulator for autonomous navigation
of robotic fishes,” In Proceedings of IEEE/RSJ International
Conference on Intelligent Robots and Systems. Sendai, Japan, pp. 613–
618, October 2004.
[2] M. Smith, D. Neumerkel and S. Hofer, “Neural networks for modelling
and control of a non-linear dynamic system,” In IEEE International
Symposium on Intelligent Control. Glasgow, Scotland, pp. 404-409,
August 1992.
[3] Hagan, M. T., H. B. Demuth and O. De Jesús, “An introduction to the
use of neural networks in control systems,” International Journal of
Robust and Nonlinear Control, vol. 12, Issue 11, pp 959–
985, September 2002.
[4 ] L. A. Chin, R. Lucio, K. Newman and S. Peugh, “Automated Zebrafish
Tracking,” University of Arizona, 2011.
[5] I. J. Leontaritis, S. A. Billings, “Input-Output Parametric Models for
non-linear systems,” Int. Journal of Control, vol. 41, No. 2, pp 329-
344, 1985.
[6] H. Demuth and M. Beale, “Neural network toolbox user's guide,”
version 4, July 2002.

Ammar I. Majeed was born in Iraq. He received his

B.Sc. degree in Electrical Engineering from University
of Technonolgy, Iraq, in 1995. He received his M.Sc.
Degree from the same University in 1998. He worked as
Assistant Lecturer in the Department of Electrical
Engineering University of Sana'a, Yemen, 1999–2000,
and then at the Department of Computer Engineering,
Al-Watinyah University, Yemen, from 2000 to 2004. He
worked as a Lecturer in the Department of Electrical Engineering at Al-
Mustansiyah University, Iraq, 2006 - till now. His field of interest is
robotics, distributed control and industrial Automation. Since 2012, he has
been pursuing a Ph.D. degree in Electrical Engineering at the University of

© 2014 GSTF
47