ABSTRACT The BLUE EYES technology aims at creating

computational machines that have perceptual

and sensory ability like those of human beings.
Is it possible to create a computer, which can It uses non-obtrusige sensing method,
interact with us as we interact each other? For employing most modern video cameras and
example imagine in a fine morning you walk microphones to identifies the users actions
on to your computer room and switch on your through the use of imparted sensory abilities .
computer, and then it tells you “Hey The machine can understand what a user
wants, where he is looking at, and even realize
friend, good morning you seem to be a bad
his physical or emotional state
mood today. And then it opens your mail box
and shows you some of the mails and tries to
cheer you. It seems to be a fiction, but it will
be the life lead by “BLUE EYES” in the very
EMOTION MOUSE
near future.
One goal of human computer interaction (HCI)
The basic idea behind this technology is to
is to make an adaptive, smart computer
give the computer the human power. We all
system. This type of project could possibly
have some perceptual abilities. That is we can
include gesture recognition, facial recognition,
understand each others feelings. For example
eye tracking, speech recognition, etc. Another
we can understand ones emotional state by
non-invasive way to obtain information about
analyzing his facial expression. If we add these
a person is through touch. People use their
perceptual abilities of human to computers
computers to obtain, store and manipulate data
would enable computers to work together with
using their computer. In order to start creating
human beings as intimate partners. The
smart computers, the computer must start
“BLUE EYES” technology aims at creating
gaining information about the user. Our
computational machines that have perceptual
proposed method for gaining user information
and sensory ability like those of human beings.
through touch is via a computer input device,
the mouse. From the physiological data
obtained from the user, an emotional state may
be determined which would then be related to
INTRODUCTION
the task the user is currently doing on the
computer. Over a period of time, a user model
Imagine yourself in a world where humans
will be built in order to gain a sense of the
interact with computers. You are sitting in
user's personality. The scope of the project is
front of your personal computer that can listen,
to have the computer adapt to the user in order
talk, or even scream aloud. It has the ability to
to create a better working environment where
gather information about you and interact with
the user is more productive. The first steps
you through special techniques like facial
towards realizing this goal are described here.
recognition, speech recognition, etc. It can
even understand your emotions at the touch of
the mouse. It verifies your identity, feels your
presents, and starts interacting with you .You 1. INTRODUCTION
ask the computer to dial to your friend at his
office. It realizes the urgency of the situation
through the mouse, dials your friend at his Blue Eyes
office, and establishes a connection.
system provides technical
Human cognition depends primarily on the
ability to perceive, interpret, and integrate means for monitoring and
audio-visuals and sensoring information.
Adding extraordinary perceptual abilities to recording the operator’s basic
computers would enable computers to work
together with human beings as intimate physiological parameters. The
partners. Researchers are attempting to add most important parameter is
more capabilities to computers that will allow
them to interact like humans, recognize human saccadic activity ,
1
which
presents, talk, listen, or even guess their
feelings. enables the system to monitor
the status of the operator’s
visual attention along with voice, physiological
head acceleration, which parameters and an overall
accompanies large view of the operating room are
displacement of the visual axis recorded. This helps to
(saccades larger than 15 reconstruct the course of
degrees).Complex industrial operators’ work and provides
environment can create a data for long-term analysis.
danger of exposing the
operator to toxic substances, Data Acquisition Unit).
which can affect his cardiac, The overall system diagram is
circulatory and pulmonary shown in Figure 1.
systems. Thus, on the grounds The tasks of the mobile Data
of plethysmographic signal Acquisition Unit are to
taken from the forehead skin maintain Bluetooth
surface, the system computes connections, to get information
heart beat rate and blood from the sensor and sending it
oxygenation. over the wireless connection, to
deliver the alarm messages sent
The Blue Eyes from the Central System Unit
system checks above to the operator and handle
parameters against abnormal personalized ID cards. Central
(e.g. a low level of blood System Unit maintains the
oxygenation or a high pulse other side of the Bluetooth
rate) or undesirable (e.g. a connection, buffers incoming
longer period of lowered sensor data, performs on-line
visual attention) values and data analysis, records the
triggers user-defined alarms conclusions for further
when necessary. Quite often in exploration and provides
an emergency situation visualization interface.
operator speak to themselves
expressing their surprise or
stating verbally the problem.
Therefore, the operator’s
 The priority of
the central unit is to provide
real-time buffering and
incoming sensor signals and
semi-real-time processing of
the data, which requires
speed-optimizes filtering and
reasoning algorithms.
Moreover, the design should
assure the possibility of
distributing the processing
among two or more central
1.1. PERFORMANCE
unit nodes (e.g. to offload the
REQUIREMENTS
database system related tasks
to a dedicated server).
The portable
nature of the mobile unit
1.2. DESIGN
results in a number of
METHODOLOGIES
performance requirements.
As the device is intended to
In creating the
run on batteries, low power
BlueEyes system a waterfall
consumption is the most
software development model
important constraint.
was used since it is suitable
Moreover, it is necessary to
for unrepeatable and
assure proper timing while
explorative projects. During
receiving and transmitting
the course of the development
sensor signals. To make the
UML standard notations were
operation comfortable the
used. They facilitate
device should be lightweight
communication between team
and electrically safe. Finally
members, all the ideas are
the use of standard and
clearly expressed by means of
inexpensive IC’s will keep
various diagrams, which is a
the price of the device at
relatively low level.
 sound base for further
development.
The results of the
functional design phase were
documented on user case
diagrams. During the low- Figure 2: Mobile Device User
level design stage the whole
systems was divided into five
main modules. Each of them
Authorization – the function is
has an independent, well-
used when the operator’s duty
defined functional interface
starts. After inserting his
providing precise description
personal ID card into the
of the services offered to the
mobile device and entering
other modules. All the
proper PIN code the device will
interfaces are documented on
start listening for incoming
UML class, interaction and
Bluetooth connections. Once
state diagrams. At this point
the connection has been
each of the modules can be
established and authorization
assigned to a team member,
process has succeeded (the PIN
implemented and tested in
code is correct) central system
parallel. The last stage of the
starts monitoring the operator’s
project is the integrated
physiological parameters. The
system testing.
authorization process shall be
repeated after reinserting the ID
card. It is not, however,
required on reestablishing
2
Bluetooth connection.
alerts. Such limited
functionality assures the device does
Receiving alerts – the function
not disturb the work of the operator
supplies the operator with the
(Fig. 2).
information about the most
important alerts regarding his
or his co-workers’ condition
and mobile device state (e.g.
connection lost, battery low). During the on-
Alarms are signaled by using a line browsing it is possible to
beeper, earphone providing watch a list of currently
central system sound feedback working operators and the
and a small alphanumeric LCD status of their mobile devices.
display, which shows more Selecting one of the operators
detailed information. enables the supervisor to check
the operator’s current
Supervisor is a person physiological condition (e.g. a
responsible for analyzing pie chart showing active brain
operators’ condition and involvement) and a history of
performance. The supervisor alarms regarding the operator.
receives tools for inspecting All new incoming alerts are
present values of the displayed immediately so that
parameters (On-line browsing) the supervisor is able to react
as well as browsing the results fast. However, the presence of
of long-term analysis (Off-line the human supervisor is not
browsing). necessary since the system
equipped with reasoning
algorithms and can trigger user-
defined actions (e.g. to inform
the operator’s co-workers).
During off-line
browsing it is possible to
reconstruct the course of the
operator’s duty with all the
physiological parameters, audio
and video data. A
comprehensive data analysis
can be performed enabling the
supervisor to draw conclusions
on operator’s overall
performance and competency saccadic activity during a
(e.g. for working night shifts). longer period of time → inform
operator’s co-workers, wake
System him up using the beeper or
administrator is a user that playing appropriate sound and
maintains the system. The log the event in the database).
administrator delivers tools for Designing new
adding new operators to the analyzer modules-based on
database, defining alarm earlier recorded data the
conditions, administrator can create new
analyzer module that can
recognize other behaviors than
those which are built-in the
system. The new modules are
created using decision tree
induction algorithm. The
administrator names the new
behavior to be recognized and
points the data associated with
it. The results received from the
new modules can be used in
alarm conditions.
configuring logging tools and
Monitoring
creating new analyzer modules.
setup enables the administrator
While registering new to choose the parameters to
operators the administrator monitor as well as the
enters appropriate data to the algorithms of the desired
system database and programs accuracy to compute parameter
his personal ID card. values.
Defining alarm Logger setup
conditions – the function provides tools for selecting the
enables setting up user-defined parameters to be recorded. For
alarm conditions by writing audio data sampling frequency
condition-action rules (e.g. low
 can be chosen. As regards the
video signal, a delay between
storing consecutive frames can
be set (e.g. one picture in every
two seconds).
Database
maintenance – here the
administrator can
remove old or
“uninteresting” data
from the database. The
“uninteresting” data is
suggested by the built-
in reasoning system.
raw digital data regarding eye
position, the level of blood
oxygenation, acceleration along
horizontal and vertical axes and
ambient light intensity. Eye
movement is measured using
direct infrared oculographic
transducers. (The eye movement
is sampled at 1 kHz, the other
parameters at 250 Hz. The sensor
sends approximately 5.2 kB of
data per second.)

2.2. DATA
ACQUISITION UNIT
(DAU)
This section deals
with the hardware part of the Blue
Eyes system with regard to the
physiological data sensor, the DAU
hardware components and the
microcontroller software.
2.2.2. HARDWARE
SPECIFICATION
2.2.1. PHYSIOLOGICAL
DATA SENSOR
Microcontrollers
To provide the Data
(e.g. Atmel 8952
Acquisition Unit with necessary
microcontroller)can be used as
physiological data an off-shelf
the core of the Data Acquisition
eye movement sensor – Jazz
Unit since it is a well-
Multisensor is used. It supplies
established industrial standard
and provides necessary power level and the state of
functionalities(i.e. high speed wireless connection.
serial port)at a low price. As the Bluetooth
The Bluetooth module communicates
module supports synchronous asynchronously with the
voice data transmission .The microcontroller there was a
codec reduces the need of implementing a cyclic
microcontroller’s tasks and serial port buffer, featuring
lessens the amount of data UART CTS/RTS flow control
being sent over the UART. The and a producer-consumer
Bluetooth module performs synchronization mechanism.
voice data compression, which
results in smaller bandwidth
when the
utilization and better sound
operators roam around the
qualit
covered area a connection with
an appropriate Bluetooth device
. is established and the data
Communication between the stream is redirected
Bluetooth module and the accordingly.
microcontroller is carried on
using standard UART interface. The data of each
The speed of the UART is set supervised operator is buffered
to 115200 bps in order to separately in the dedicated
assure that the entire sensor Operator Manager. At the
data is delivered in time to the startup it communicates with
central system. the Operator Data Manager in
The order to get more detailed
alphanumeric LCD display personal data. The most
gives more information of important Operator Manager’s
incoming events and helps the task is to buffer the incoming
operator enter PIN code. The raw data and to split it into
LED indicators shows the separate data streams related to
result of built-in-self-test, each of the measured
parameters. The raw data is measuring the time delay
sent to a Logger Module, the between subsequent extremes
split data streams are available of one of the signals. We
for the other system modules decided not to process only one
through producer-consumer of the data streams – the
queues. Furthermore, the algorithm is designed to choose
Operator Manager provides an dynamically one of them on the
interface for sending alert grounds of the signal level.
messages to the related Unfortunately, the both signals
operator. are noised so they must be
information. filtered before further
processing. We considered a
number of different algorithms
and decided to implement
average value based smoothing.
More detailed discussion is
presented in section 3.3.5
Tradeoffs and Optimization.
The algorithm consists in
calculating an average signal
value in a window of 100
Figure 9: samples. In every step the
Saccade occurrence and window is advanced 5 samples
Visual attention level in order to reduce CPU load.
This approach lowers the
The Pulse rate sampling rate from 250 Hz
analyzer registers for the down to 50 Hz. However, since
oxyhemoglobin and the Visual heartbeat frequency
deoxyhemoglobin level data is at most 4 Hz the Nyquist
streams. Since both signals condition remains satisfied.
contain a strong sinusoidal The figures show the signal
component related to heartbeat, before (Fig. 10a) and after
the pulse rate can be calculated filtering (Fig 10b).
 Additionally, we
implemented a simple module
that calculates average blood

Figure: 10(a) oxygenation level. Despite its

Figure: simplicity the parameter is an

10(b) important measure of the

operator’s physiological
condition.
After filtering
the signal the pulse calculation
algorithm is applied. The The other signal

algorithm chooses the point to features that are not recognized

be the next maximum if it by the built-in analyzers can be

satisfies three conditions: extracted using custom

points on the left and on the modules created by Decision

right have lower values, the Tree

previous extreme was a Induction module. The custom module

processes the generated decision tree,
minimum, and the time
registers for needed data streams and
between the maximums is not produces the desired output signal.
too short (physiological
constraint). The new pulse
Decision Tree
value is calculated based on the
Induction module generates the
distance between the new and
decision trees, which are binary
the previous maximum
trees with an attribute test in
detected. The algorithm gets
each node. The decision tree
the last 5 calculated pulse
input data is an object
values and discards 2 extreme
described by means of a set of
values to average the rest.
attribute-value pairs. The
Finally, it does the same with
algorithm is not able to process
the minimums of the signal to
time series directly. The
obtain the second pulse rate
attributes therefore are average
value, which gives the final
signal value, signal variance
result after averaging.
and the strongest sinusoidal
components. As an output the
decision tree returns
category the object belongs to.
In the Decision Tree Induction
module we mainly use C 4.5
algorithm [2], but also propose
our own modifications. The
algorithm is a supervised
learning from examples i.e. it
considers both attributes that
describe the case and a correct
answer. The main idea is to use
a divide-and-conquer approach
to split the initial set of
examples into subsets using a
simple rule (i-th attribute less
than a value). Each division is
based on entropy calculation –
the distribution with the lowest
entropy is
Additionally, we propose many
modifications concerning some
steps of the algorithm and
further exploration
system.
the
For each case to
be classified C 4.5 traverses the
tree until reaching the leaf
where appropriate category id
chosen. is stored. To increase the hit
ratio our system uses more
advanced procedure. For single
analysis we develop a group of
of the k trees (where k is a
parameter), which we call a
decision forest. Initial example
set S is divided randomly into
k+1 subsets S0 ... Sk. S0 is left
to test the whole decision
forest. Each tree is induced
using various modifications of
the algorithm to provide
results’ independence. Each i-
th tree is taught using S\S0\Si 2.3. TOOLS USED TO
set (S without S0 and Si sets) DEVELOP
and tested with Si that estimates
BLUEEYES
a single tree error ratio.
Furthermore we extract all
In creating the
wrongly classified examples
hardware part of the DAU a
and calculate correlation matrix
development board was built,
between each pair of the trees.
which enabled to mount,
In an exploring phase we use
connect and test various
unequal voting rule – each tree
peripheral devices cooperating
has a vote of strength of its
with the microcontroller.
reliability. Additionally, if two
During the implementation of
trees give the same answer their
the DAU there was a need for a
vote is weakened by the
piece of software to establish
correlation ratio.
and test Bluetooth connections.
Hence created a tool called
Alarm
Blue Dentist. The tool
Dispatcher Module is a very
provides support for controlling
important part of the Data
the currently connected
Analysis module. It registers
Bluetooth device. Its functions
for the results of the data
are: Local device management
analysis, checks them with
(resetting, reading local
regard to the user-defined
BD_ADDR, putting in
alarm conditions and launches
Inquiry/Page and Inquiry/Page
appropriate actions when
scan modes, reading the list of
needed. The module is a
locally supported features and
producer of the alarm
setting UART speed) and
messages, so that they are
connection management
accessible in the logger and
(receiving and displaying
visualization modules.
Inquiry scan results,
establishing ACL links, adding
SCO connections, performing
link authorization procedure,
sending test data packets and
disconnecting).
stream. They used the software
to measure database system
performance and to optimize
some of the SQL queries (e.g.
we replaced correlated SQL
queries with cursor operations).

Fig:
Blue Dentist

To test the
possibilities and performance
of the remaining parts of the
Project Kit (computer, camera Figure

and database software) Blue 12: Blue Capture

Capture (Fig. 12) was Also a simple

created. The tool supports tool for recording Jazz

capturing video data from Multisensory measurements

various sources (USB web- was introduced. The program

cam, industrial camera) and reads the data using a parallel

storing the data in the MS SQL port and writes it to a file. To

Server database. Additionally, program the operator’s personal

the application performs sound ID card we use a standard

recording. After filtering and parallel port, as the EEPROMs

removing insignificant and the port are both TTL-

fragments (i.e. silence) the compliant. A simple dialog-

audio data is stored in the based application helps to

database. Finally, the program accomplish the task.

plays the recorded audiovisual

3. SUMMARY

The BlueEyes
system is developed because of
the need for a real-time
monitoring system for a human
operator. The approach is

REFERENCE

1. Carpenter R. H. S.,
Movements of the eyes, 2nd
edition, Pion Limited, 1988,
London.
2. Bluetooth specification,
version 1.0B, Bluetooth SIG,
1999.
3. ROK 101 007 Bluetooth
Module ,Ericsson
Microelectronics,2000.
4. AT89C52 8-bit
Microcontroller Datasheet,
Atmel.
5. Intel Signal Processing
Library –Reference Manual.