You are on page 1of 24

Computer Networks 34 (2000) 547±570

www.elsevier.com/locate/comnet

Intrusion detection using autonomous agents


Eugene H. Spa€ord, Diego Zamboni *
Center for Education and Research in Information Assurance and Security, 1315 Recitation Building, Purdue University, West Lafayette,
IN 47907-1315, USA

Abstract
AAFID is a distributed intrusion detection architecture and system, developed in CERIAS at Purdue University.
AAFID was the ®rst architecture that proposed the use of autonomous agents for doing intrusion detection. With its
prototype implementation, it constitutes a useful framework for the research and testing of intrusion detection algo-
rithms and mechanisms. We describe the AAFID architecture and the existing prototype, as well as some design and
implementation experiences and future research issues. Ó 2000 Elsevier Science B.V. All rights reserved.

Keywords: Intrusion detection; Software agents; Distributed systems; Security; Perl

1. Introduction puter system without authorization (i.e., `crack-


ers') and those who have legitimate access to the
The intrusion detection ®eld has grown con- system but are abusing their privileges (i.e.,
siderably in the last few years, and a large number the `insider threat')'' [25]. We add to this de®nition
of intrusion detection systems have been developed the identi®cation of attempts to use a computer
to address di€erent needs. Intrusion detection is system without authorization or to abuse existing
clearly necessary with the growing number of privileges. Our working de®nition matches the one
computer systems being connected to networks. given by Heady et al. [15], where an intrusion
We describe an architecture for intrusion detection is de®ned as ``any set of actions that attempt to
and system monitoring based on autonomous compromise the integrity, con®dentiality, or
agents that serves as a research framework for availability of a resource'', disregarding the success
intrusion detection techniques and algorithms. or failure of those actions.
We start by de®ning some common terms and The dictionary de®nition of the word intrusion
the motivation for using autonomous agents in an [24] does not include the concept of an insider
intrusion detection system. abusing his or her privileges to perform unautho-
rized actions, or attempting to do so. A more ac-
1.1. Intrusion detection curate phrase to use is intrusion and insider abuse
detection. In this document we use the term in-
Intrusion detection is de®ned as ``the problem trusion to mean both intrusion and insider abuse.
of identifying individuals who are using a com- An intrusion detection system is a computer
system (possibly a combination of software and
*
Corresponding author.
hardware) that attempts to perform intrusion de-
E-mail addresses: spaf@cerias.purdue.edu (E.H. Spa€ord), tection, as de®ned above. Most intrusion detection
zamboni@cerias.purdue.edu (D. Zamboni). systems try to perform their task in real time [25],
1389-1286/00/$ - see front matter Ó 2000 Elsevier Science B.V. All rights reserved.
PII: S 1 3 8 9 - 1 2 8 6 ( 0 0 ) 0 0 1 3 6 - 5
548 E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570

but there are also intrusion detection systems that available can cause changes in system use pat-
do not operate in real time, either because of the terns.
nature of the analysis they perform (e.g., [20]) or As the number of systems to be monitored in-
because they are geared for forensic analysis [13,32]. creases and the chances of attacks increase we also
Intrusion detection systems are usually classi- consider the following characteristics as desirable:
®ed as host-based or network-based [25]. Host- · It must be scalable to monitor a large number
based systems base their decisions on information of hosts while providing results in a timely and
obtained from a single host (usually audit trails), accurate manner.
while network-based systems obtain data by · It must provide graceful degradation of service.
monitoring the trac in the network to which the If some components of the intrusion detection
hosts are connected. system stop working for any reason, the rest of
The de®nition of an intrusion detection system them should be a€ected as little as possible.
does not include preventing the intrusion from · It must allow dynamic recon®guration, allowing
occurring, only detecting it and reporting it to an the administrator to make changes in its con®g-
operator. There are some intrusion detection sys- uration without the need to restart the whole
tems (for example, [6]) that try to react when they intrusion detection system.
detect an unauthorized action occurring. This re-
action usually includes trying to contain or stop 1.3. Distributed and centralized intrusion detection
the damage, for example, by terminating a net- systems
work connection.
Intrusion detection systems are also usually
classi®ed by the way their components are dis-
1.2. Desirable characteristics of an intrusion detec- tributed
tion system · A centralized intrusion detection system is one
where the analysis of the data is performed in
Crosbie and Spa€ord [10] de®ned the following a ®xed number of locations, independent of
characteristics as desirable for an intrusion detec- how many hosts are being monitored. We do
tion system: not consider the location of the data collection
· It must run continually with minimal human components, only the location of the analysis
supervision. components. Some intrusion detection systems
· It must be fault tolerant by being able to recover that we classify as centralized are: IDES
from system crashes, either accidental or caused [11,12,22,23], IDIOT [7,21], NADIR [17] and
by malicious activity. Upon startup, the intrusion NSM [16].
detection system must be able to recover its pre- · A distributed intrusion detection system is one
vious state and resume its operation una€ected. where the analysis of the data is performed in
· It must resist subversion. The intrusion detection a number of locations proportional to the num-
system must be able to monitor itself and detect ber of hosts that are being monitored. Again, we
if it has been modi®ed by an attacker. only consider the locations and number of the
· It must impose a minimal overhead on the sys- data analysis components, not the data collec-
tems where it runs, to avoid interfering with tion components. Some intrusion detection sys-
the systems normal operation. tems that we classify as distributed are: DIDS
· It must be con®gurable to accurately implement [29,30], GrIDS [4,31], EMERALD [26] and
the security policies of the systems that are being AAFID [1].
monitored.
· It must be adaptable to changes in system and 1.3.1. How centralized and distributed intrusion
user behavior over time. For example, new ap- detection systems compare
plications being installed, users changing from Table 1 comments on the advantages and dis-
one activity to another or new resources being advantages of centralized and distributed intrusion
E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570 549

detection systems with respect to the desirable 1.4. Autonomous agents


characteristics described in Section 1.2.
We can see that centralized intrusion detection A software agent can be de®ned as [3]:
systems have some advantages over distributed
intrusion detection systems, but these advantages . . .a software entity which functions continu-
are not insurmountable through technical means. ously and autonomously in a particular envi-
Centralized intrusion detection systems, however, ronment. . .able to carry out activities in a
have some fundamental limitations, such as their ¯exible and intelligent manner that is respon-
lack of scalability and the diculty to provide sive to changes in the environment. . .Ideally,
graceful degradation of service. The intrusion de- an agent that functions continuously. . .would
tection ®eld has been shifting in the last few years be able to learn from its experience. In addition,
towards designing and building distributed intru- we expect an agent that inhabits an environ-
sion detection systems (e.g., [1,2,18,26,31]). In this ment with other agents and processes to be able
paper, we discuss one approach to building such a to communicate and cooperate with them, and
system using autonomous agents. perhaps move from place to place in doing so.

Table 1
Comparison between centralized and distributed intrusion detection systems with respect to the desirable characteristics described in
Section 1.2
Characteristic Centralized Distributed
Run continually A relatively small number of components need to Harder because a larger number of components
be kept running. need to be kept running.
Fault tolerant The state of the intrusion detection system is The state of the intrusion detection system is
centrally stored, making it easier to recover it after distributed, making it more dicult to store in a
a crash. consistent and recoverable manner.
Resist subversion A smaller number of components need to be A larger number of components need to be
monitored. However, these components are larger monitored. However, because of the larger
and more complex, making them more dicult to number, components can cross-check each other.
monitor. The components are also usually smaller and less
complex.
Minimal overhead Impose little or no overhead on the systems, Impose little overhead on the systems because the
except for the ones where the analysis components components running on them are smaller.
run, where a large load is imposed. Those hosts However, the extra load is imposed on most of
may need to be dedicated to the analysis task. the systems being monitored.
Con®gurable Easier to con®gure globally, because of the smaller Each component may be localized to the set of
number of components. It may be dicult to tune hosts it monitors, and may be easier to tune to its
for speci®c characteristics of the di€erent hosts speci®c tasks or characteristics.
being monitored.
Adaptable By having all the information in fewer locations, it Data are distributed, which may make it more
is easier to detect changes in global behavior. dicult to adjust to global changes in behavior.
Local behavior is more dicult to analyze. Local changes are easier to detect.
Scalable The size of the intrusion detection system is A distributed intrusion detection system can scale
limited by its ®xed number of components. As the to a larger number of hosts by adding components
number of monitored hosts grows, the analysis as needed. Scalability may be limited by the need
components will need more computing and to communicate between the components, and by
storage resources to keep up with the load. the existence of central coordination components.
Graceful degradation If one of the analysis components stops working, If one analysis component stops working, part of
of service most likely the whole intrusion detection system the network may stop being monitored, but the
stops working. Each component is a single point rest of the intrusion detection system can continue
of failure. working.
Dynamic A small number of components analyze all the Individual components may be recon®gured and
recon®guration data. Recon®guring them likely requires the restarted without a€ecting the rest of the intrusion
intrusion detection system to be restarted. detection system.
550 E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570

For our purposes, we de®ne an autonomous bility is to have agents do cross-veri®cation, with
agent (henceforth agent) as a software agent that each agent being checked periodically by several
performs a certain security monitoring function at others.
a host. If an agent collects network information related
We term the agents as autonomous because they to the host where it is running, we reduce the
are independently-running entities (i.e., their exe- possibility of insertion and evasion attacks [27],
cution is scheduled only by the operating system, which are also a form of subversion, and to which
and not by another process). Agents may or may network-based intrusion detection systems are
not need data produced by other agents to per- usually subject.
form their work. Additionally, agents may receive Minimal overhead: Well-designed agents can
high-level control commands ± such as indications impose a minimal load on the system where they are
to start or stop execution, or to change some op- running. Furthermore, agents can be enabled
erating parameters ± from other entities. Neither and disabled dynamically, making it possible to
of these characteristics detriments our de®nition of use resources only for the tasks needed at each
agent autonomy. moment.
Con®gurable: An agent can be con®gured (or
1.4.1. Using autonomous agents to build a better even implemented) speci®cally for the needs of the
distributed intrusion detection system host where it will run. Autonomous agents there-
Because agents are independently-running enti- fore provide the possibility for ®ne-grained con-
ties, they can be added, removed and recon®gured ®guration abilities.
without altering other components and without Adaptable: Each agent has local knowledge that
having to restart the intrusion detection system. makes it able to adapt to changes in local behav-
Agents can be tested on their own before intro- ior. By having a higher-level view of the system
ducing them into a more complex environment. An state, it may be possible to adapt to changes in
agent may also be part of a group that performs global behavior as well. One way of adapting is to
di€erent simple functions but that can exchange automatically add or remove agents to monitor
information to derive more complex results things that are deemed interesting at a certain
than any one of them may be able to obtain on their point in time.
own. Scalable: Agents are deployed to the hosts that
We analyze the performance of an intrusion need to be monitored. Therefore, an intrusion
detection system built using autonomous agents detection system built using agents can grow
with respect to the desirable characteristics listed simply by deploying new agents as needed. The
in Section 1.2: bottleneck may be the communication mecha-
Continuous running: If an intrusion detection nisms between the agents and the central coor-
system is constituted of a number of autonomous dination components, if they exist, as well as the
agents, some of them may be taken o€-line for processing capabilities of those components.
maintenance or for other reasons while others Both of these problems can be solved, as has
keep running, therefore providing intrusion de- been shown by proposed schemes to build in-
tection functionality on a continuous manner. trusion detection systems without the need of a
Fault tolerance: The storage and recovery of central controlling entity [18,34] and that mini-
global state is still problematic, as described in mize communication needed between components
Section 1.3 for distributed intrusion detection [18].
systems. Autonomous agents, however, would Even when using traditional communication
be able to keep local state and recover it upon schemes, the intrusion detection system can be
startup. made scalable by organizing the agents in a hier-
Resist subversion: Self-monitoring in autono- archical structure. This idea was proposed by
mous agents is a dicult problem, but it is also the Crosbie and Spa€ord [9] and was also used by
subject of current research (e.g., [14]). One possi- Staniford-Chen et al. [31].
E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570 551

Graceful degradation of service: If one agent · If agents are implemented as separate processes
stops working for any reason, one or two things on a host, each agent can be implemented in the
may happen programming language that is best suited for the
· If the agent produces results on its own, only its task that it has to perform.
results will be lost. All other agents will continue
to work normally.
· If the data produced by the agent was needed by 2. The AAFID architecture
other agents, that group of agents may be im-
peded from working properly. Even in this case, We propose an architecture called autonomous
the data dependencies between agents are agents for intrusion detection (AAFID) for
known in advance, so the consequences of fail- building intrusion detection systems that use
ure can be predicted. agents as their lowest-level element for data col-
In any case, the damage is restricted to at most lection and analysis.
a set of agents. Thus, if the agents are properly
organized in mutually independent sets, the de- 2.1. History of the AAFID architecture
gradation of the service provided by the intrusion
detection system will be gradual and proportional What was to become the AAFID architecture
to the number of agents that stop functioning. was ®rst proposed by Crosbie and Spa€ord in 1994
Dynamic recon®guration: The ability to start [8]. In this paper, the authors proposed (for the
and stop agents independently of each other ®rst time in the published literature) the idea of
creates the possibility of recon®guring the intru- using autonomous agents for performing intrusion
sion detection system without having to restart it. detection, and suggested that the agents could be
If we need to start collecting a new type of data or evolved automatically using genetic programming
monitoring for a new kind of attacks, the appro- so that the intrusion detection system would au-
priate agents can be started without disturbing the tomatically adjust and evolve according to user
ones that are already running. Similarly, agents behavior.
that are no longer needed can be stopped, and The idea of using genetically programmed
agents that need to be recon®gured can be sent the agents was never fully implemented and tested.
appropriate commands without having to restart However, the idea of using agents for intrusion
the whole intrusion detection system. detection kept evolving, and between 1995 and
Additional bene®ts: Using agents as data col- 1996 the AAFID architecture was developed in the
lection and analysis entities also provides the fol- COAST Laboratory. The initial architecture had
lowing desirable features: the hierarchical structure that remains to date,
· Because an agent can be programmed arbitrari- included monitors, transceivers and agents, and
ly, it can obtain its data from an audit trail, by was used to implement the ®rst prototype of the
probing the system where it is running, by cap- system.
turing packets from a network, or from any oth- From 1997 to date the AAFID architecture
er suitable source. Thus, an intrusion detection evolved with the addition of ®lters and the sepa-
system built using agents can cross the tradition- ration of the user interface from the monitor. The
al boundaries between host-based and network- new architecture has been used for the implemen-
based intrusion detection systems. tation of the latest prototype.
· Because agents can be stopped and started with-
out disturbing the rest of the intrusion detection 2.2. Overview
system, agents can be upgraded as increased
functionality is required from them. As long as A simple example of an intrusion detection
their external interface remains compatible, oth- system that adheres to the AAFID architecture is
er components need not even know that the shown in Fig. 1(a). This ®gure shows the four
agent has been upgraded. components of the architecture: agents, ®lters,
552 E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570

Fig. 1. Physical and logical representations of a sample intrusion detection system that follows the AAFID architecture (called an
AAFID system). (a) Physical layout of the components in a sample AAFID system, showing agents, ®lters, transceivers and monitors,
as well as the communication and control channels between them. (b) Logical organization of the same AAFID showing the com-
munication hierarchy of the components. The bidirectional arrows represent both the control and data ¯ow between the entities.
Notice that the logical organization is independent of the physical location of the entities in the hosts.

transceivers and monitors. We refer to each one of interesting events occurring in the host. Agents
these components as AAFID entities or simply may use ®lters to obtain data in a system-inde-
entities, and to the whole intrusion detection sys- pendent manner. All the agents in a host report
tem constituted by them as an AAFID system. their ®ndings to a single transceiver. Transceivers
An AAFID system can be distributed over any are per-host entities that oversee the operation of
number of hosts in a network. Each host can all the agents running in their host. They have
contain any number of agents that monitor for the ability to start, stop and send con®guration
E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570 553

commands to agents. They may also perform data of telnet connections within the last 5 min, which is
reduction on the data received from the agents. an existing agent in the current AAFID imple-
The transceivers report their results to one or mentation), or a complex software system (for
more monitors. Each monitor oversees the opera- example, an instance of IDIOT [7] looking for a set
tion of several transceivers. Monitors have access of local intrusion patterns). As long as the agent
to network-wide data, therefore they are able to produces its output in the appropriate format and
perform higher-level correlation and detect intru- sends it to the transceiver, it can be part of the
sions that involve several hosts. Monitors can be AAFID system.
organized in a hierarchical fashion such that a Agents may perform any functions they need.
monitor may in turn report to a higher-level Some possibilities (which have not been used by
monitor. Also, a transceiver may report to more any existing AAFID agents) are:
than one monitor to provide redundancy and re- · Agents may evolve over time using genetic pro-
sistance to the failure of one of the monitors. Ul- gramming techniques, as suggested in [9].
timately, a monitor is responsible for providing · Agents may employ techniques to retain state
information and getting control commands from a between sessions, allowing them to detect long-
user interface. Fig. 1(b) shows the logical organi- term attacks or changes in behavior. Currently,
zation corresponding to the physical distribution the architecture does not specify any mecha-
depicted in Fig. 1(a). nisms for maintaining persistent state.
We now describe each component in greater · Agents could migrate from host to host by com-
detail. bining the AAFID architecture with some exist-
ing mobile-agent architecture.
2.3. Components of the architecture Agents can be written in any programming lan-
guage. Some functionality (e.g., reporting, com-
2.3.1. Agents munication and synchronization mechanisms) is
An agent is an independently-running entity common to all the agents, and can be provided
that monitors certain aspects of a host, and reports through shared libraries or similar mechanisms.
to the appropriate transceiver. For example, an Thus, a framework implementation, like the one
agent could be looking for a large number of telnet described in Section 3, can provide most of the
connections to a protected host, and consider their tools and mechanisms necessary to make writing
occurrence as suspicious. The agent would gener- new agents a relatively simple task.
ate a report that is sent to the appropriate trans-
ceiver. The agent does not have the authority to 2.3.2. Filters
directly generate an alarm. Usually, a transceiver Filters are intended to be both a data selection
or a monitor will generate an alarm for the user and a data abstraction layer for agents.
based on information received from agents. By In the original AAFID architecture, each agent
combining the reports from di€erent agents, was responsible for obtaining the data it needed.
transceivers build a picture of the status of their When the ®rst prototype was implemented, this
host, and monitors build a picture of the status of approach showed the following problems:
the network they are monitoring. · On a single system, there may be more than
Agents do not communicate directly with each one agent that needs data from the same data
other in the AAFID architecture. Instead, they source. This is common in Unix with multi-
send all their messages to the transceiver. The function log ®les (such as /var/adm/mes-
transceiver decides what to do with the informa- sages). Having each agent read the data on
tion based on agent con®guration information. its own meant duplicating the work of reading
The architecture does not specify any require- the ®le, parsing it and discarding unnecessary
ments or limitations for the functionality of an records.
agent. It may be a simple program that monitors a · There may be agents that can provide a useful
speci®c event (for example, counting the number function under di€erent versions of Unix, or
554 E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570

even under di€erent architectures (such as Win- · Receives reports generated by the agents run-
dows NT). However, the data needed by the ning in its host.
agent may be located in di€erent places in each · Does appropriate processing on the information
system and may be stored in di€erent formats. received from agents.
This meant having to write a di€erent agent · Distributes the information received from the
for each system, that knows where to ®nd the agents, or the results of processing it, either to
data and how to read it. other agents or to a monitor, as appropriate.
Both of these problems are solved through the
introduction of ®lters. Filters provide a subscrip- 2.3.4. Monitors
tion-based service to agents, and have two func- Monitors are the highest-level entities in the
tions. AAFID architecture. They have control and data
Data selection: There exists only one ®lter per processing roles that are similar to those of the
data source, and multiple agents can subscribe to transceivers. The main di€erence is that monitors
it. When an agent subscribes to a ®lter, it speci®es can control entities that are running in several
which records it needs (using some criteria like di€erent hosts whereas transceivers only control
regular expressions), and the ®lter only sends to local agents.
the agent records that match the given criteria. In their data processing role, monitors receive
This eliminates duplicate work in reading and ®l- information from all the transceivers they control,
tering data. and can do higher-level correlations and detect
Data abstraction layer: Filters implement all events that involve several di€erent hosts. Moni-
the architecture- and system-dependent mecha- tors have the capability to detect events that may
nisms for obtaining the data that agents need. be unnoticed by the transceivers.
Therefore, the same agent can run under di€erent In their control role, monitors can receive in-
architectures simply by connecting to the appro- structions from other monitors and they can
priate ®lter. This makes it easier to reuse code control transceivers and other monitors. Monitors
and to run AAFID under di€erent operating have the ability to communicate with a user in-
systems. terface and provide the access point for the whole
AAFID system. Monitors implement an interface
2.3.3. Transceivers that includes mechanisms for accessing the infor-
Transceivers are the external communications mation that the monitor has, for providing com-
interface of each host. They have two roles: con- mands to the monitor, or to send commands to
trol and data processing. For a host to be moni- lower-level entities such as transceivers and
tored by an AAFID system, there must be a agents.
transceiver running on that host. If two monitors control the same transceiver,
In its control role, a transceiver performs the mechanisms have to be employed to ensure con-
following functions: sistency of information and behavior. The AAFID
· Keeps track and controls execution of agents in architecture does not currently specify the mech-
its host. The instructions to start and stop anisms for achieving this consistency.
agents can come from con®guration informa-
tion, from a monitor, or as a response to speci®c 2.3.5. User interfaces
events (for example, a report from one agent The most complex and feature-full intrusion
may trigger the activation of other agents to per- detection system can be useless if it does not have
form a more detailed monitoring of the host). good mechanisms for users to interact with it.
· Responds to commands issued by its monitor by The AAFID architecture clearly separates the
providing the appropriate information or per- user interface from the data collection and pro-
forming the requested actions. cessing elements. A user interface has to interact
In its data processing role, a transceiver has the with a monitor to request information and to
following duties: provide instructions.
E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570 555

This separation allows di€erent user interface it enables us to test ideas on real situations. In this
implementations to be used with an AAFID sys- section we describe the objectives, design and
tem. For example, a graphical user interface implementation decisions that have been made in
(GUI) could be used to provide interactive access the AAFID implementations.
to the intrusion detection system, while a com-
mand-line based interface could be used in scripts 3.1. History of implementations
to automate some maintenance and reporting
functions. The ®rst AAFID prototype was implemented
between 1995 and 1996, based on the ®rst speci®-
2.4. Communication mechanisms cation of the AAFID architecture. This prototype
was implemented by a combination of programs
The transmission of messages between entities is written in C, Bourne shell, AWK and Perl. Its
a central part of the functionality of an AAFID main objective was to ``put something together'' to
system. Although the AAFID architecture does test the initial feasibility of the architecture, and to
not specify which communication mechanisms get some feedback on design and implementation
have to be used, we consider the following to be decisions. This ®rst implementation was only
some important points about the communication tested internally in the COAST Laboratory.
mechanisms used in an AAFID system: The second implementation, which has evolved
· Appropriate mechanisms should be used for dif- into the current version of the AAFID prototype,
ferent communication needs. In particular, com- was started in late 1997, and has evolved to in-
munication within a host may be established by corporate the more recent changes in the archi-
di€erent means than communication across the tecture, such as ®lters. This implementation is
network. referred to as the AAFID2 prototype. In Septem-
· The communication mechanisms should be e- ber 1998, AAFID2 was released for the ®rst time to
cient and reliable in the sense that they should the public. The ®rst release included the basic
(a) not add signi®cantly to the communications system, a few agents, and had been tested under
load imposed by regular host activities, and (b) Solaris.
provide reasonable expectations of messages In September 1999, a second public release was
getting to their destination quickly and without made. The main change in the new release was the
alterations. introduction of a new event-processing mechanism
· The communication mechanisms should be se- (see Section 3.4.5). The new release also included
cure in the sense that they should (a) be resistant improved development support, and was tested
to attempts of rendering it unusable by ¯ooding under Linux as well as Solaris.
or overloading, and (b) provide some kind of The latest version of AAFID2 is described in the
authentication and con®dentiality mechanism. following sections.
The topics of secure communications, secure dis-
tributed computation and security in autonomous 3.2. Objectives of the current prototype
agents have been already studied [14,19], and
possibly some previous work can be used in The AAFID2 prototype was designed with a set
AAFID implementations to obtain communica- of speci®c objectives in mind.
tion channels that provide the necessary charac- Road-test the architecture. The AAFID archi-
teristics. tecture seems adequate for an intrusion detection
system from a design point of view but we are
interested in getting feedback from its use in real
3. The AAFID implementation situations, in real networks, and facing real attacks
and problems. The main objective in the develop-
An implementation of the AAFID architecture ment of AAFID2 was to be able to use it and ship
has been an important part of the project because it to people and organizations that can put it to
556 E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570

use, evaluate both the architecture and the imple- having to worry about the architectural mecha-
mentation, and provide feedback that enables us nisms.
to identify weaknesses and possibilities for im-
provement. 3.3. Design notes
Usability. One of our high priorities was to
make it as easy to use as possible. The di€erent In this section, we describe some of the design
modules are as independent as possible, and each decisions that we took for the development of the
one of them has a de®ned interface that allows AAFID2 prototype.
users to run and interact with it easily.
Con®gurability and extensibility. Another pri- 3.3.1. Communication model
ority was to make it easy to con®gure the behavior The logical organization of the AAFID archi-
of the components of AAFID2 , even at run time. tecture (see Fig. 1(b)) maps directly to a hierar-
The modules themselves are easy to con®gure, and chical model of communication. This model is
the overall structure of the system (including the shown in Fig. 2, and based on it we make the
layout of modules in the monitored systems, following terminology de®nitions:
and their interconnections) is easy to specify and
set up. De®nition 1 (Upstream, downstream, above and
Minimum installation requirements. Many com- below). From a given level in the AAFID com-
ponents of AAFID2 are likely to change frequently munication model as depicted in Fig. 2, we iden-
during the development and testing phases. For tify levels that are closer to the root of the
this reason, we tried to make it possible to provide hierarchy as being upstream, and levels that are
a minimum set of programs that have to be pre- closer to the leaves of the tree as downstream. One
installed in the monitored hosts, and have all the entity is above another one if it can be reached by
other modules distributed automatically when they following communication paths upstream, and it is
are needed. below if it can be reached by following communi-
No focus on performance. Although we want cation paths downstream. For example, in Fig. 2,
AAFID2 to be as ecient as possible, our focus entity A is above entities B, C, D and E; entities D
during its development was not in performance, and E are below entities A and B, but not below
but on identifying and evaluating the characteris- entity C, because there is no upstream-only or
tics that we want in an AAFID system. Once these downstream-only communication path from C to
features are set, a more ecient implementation D and E.
can be done based on them.
No focus on agent security. The use of autono-
mous agents poses security concerns, and ensuring
the integrity and authenticity of an agent is a dif-
®cult problem. We decided not to address this
problem in the implementation of the AAFID2
prototype, because it is a subject of current re-
search [14], and it does not a€ect the detection
capabilities of the AAFID architecture (although
it a€ects its applicability as a production system).
Provide an infrastructure for development. AA-
FID2 provides a simple intrusion detection system,
but more importantly it provides the infrastructure
for the implementation of more complex systems
by allowing the implementer to use all the under- Fig. 2. Model of communication in AAFID2 . The arrows
lying facilities (e.g., communications and syn- represent the communication channels (both for data ¯ow and
chronization) to build complex features without control) between the entities.
E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570 557

De®nition 2 (Super-entities and sub-entities). With that is used to control the whole intrusion detec-
respect to a speci®c entity, sub-entities are those tion system.
that are below it, and super-entities are those that
are above it. For example, in Fig. 2, entity A is a 3.3.2. Functionality needed for each entity
super-entity of all the others and entities D and E One of the ®rst phases in the development of
are sub-entities of both A and B. AAFID2 was the identi®cation of the functionality
expected from each type of entity, as well as their
De®nition 3 (Parent and child entities). For any communication requirements.
given entity, a super-entity that is in the level im-
mediately above it is called a parent entity. Simi- 3.3.2.1. All entity types. All entities in AAFID2
larly, a sub-entity that is in the level immediately must be independently-running programs. It must
below it is called a child entity. For example, in be possible for a user to start an entity from the
Fig. 2, entity A is the parent of both C and B, command line and interact with it by giving mes-
which are its children. sages and commands from the keyboard, but it
must also be possible for the entity to be started by
its controller entity.
De®nition 4 (Up and down). When an entity sends All entities have a unique identi®er and a de-
a message to an entity that is above it, we say that scription. The identi®er must provide a unique
it sends the message up. If it sends a message to an way of referencing an entity. The description must
entity that is below it, we say that the message is be a brief human-readable description of the entity
sent down. and its functionality.
All entities must react adequately to a set of
De®nition 5 (Controller entity). A controller entity messages that allow basic operations such as
is one that can exert control over a set of other stopping the entity and querying it for informa-
entities. Usually the controller of an entity is its tion. Additionally, each entity may de®ne its own
parent entity. commands for implementing speci®c functionality.
All entities must be able to exchange messages
In this model, we can identify layers of entities, with their controller entity. Any messages received
with the following properties: from the controller entity must be processed as soon
· There is a single node at the top of the hierarchy as possible. The controller entity may be in the same
(the root). host (for example, a transceiver is the control-
· An entity can only have direct communication ler entity for its agents), or in a di€erent host (for
with entities immediately above and below its example, a monitor is the controller entity for the
own. The only exception to this rule is the com- transceivers). Ideally, the implementer of an
munication between agents and ®lters, which entity should not have to worry about whether the
occurs between entities in the same layer. communication channel is local or over the
· An entity can only have one parent, but it can network.
have multiple children. We also decided that AAFID2 should include
· Agents and ®lters are in the leaves of the tree, facilities in the entity infrastructure for debugging
and thus can only send messages up. Agents and generation of log messages.
and ®lters can also send messages between them-
selves, but only when they are siblings (both are 3.3.2.2. Agents. Agents collect information from
children of the same transceiver). the system where they run and monitor for speci®c
· The entity in the root of the tree is always a situations that may indicate a security problem.
monitor. However, monitors can also appear The general behavior of an AAFID2 agent is de-
in intermediate levels. scribed by Algorithm 1. This structure allows
· The only entity that may be above the root mon- agents to react to di€erent types of events, in-
itor is a user interface, or some other program cluding timing, ®le and signal events.
558 E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570

Algorithm 1. Generic behavior of an agent. Lines (see Fig. 1(b)). Filters are started by transceivers,
in italics represent sections that have to be pro- but receive commands from and provide data to
vided by the author of the agent. agents directly. Therefore, a ®lter needs to be able
{Instantiation of the agent} to receive control data from two channels (not
{Instantiation-time initialization} only one, as all other entities), and must be able to
Set event handlers provide data through a ``side channel'' that is
{Execution of the agent} di€erent from the normal communication link to
Run-time initialization its controlling entity.
Set more event handlers Filters are data sources. Therefore, they must
Contact necessary ®lters have facilities for accessing ®les and other sources
Enter Event loop of data on the system where they run. On occasion,
a ®lter may need to run with increased privileges to
In the initial release of AAFID2 , agents had a be able to access system data.
much stricter structure based on a polling mecha-
nism. This structure is shown in Algorithm 2. The 3.3.2.4. Transceivers. Transceivers are in charge of
polling structure is sucient for many agents, controlling all the agents running in a host.
which periodically monitor for some activity or Therefore, they do not need remote communica-
event, and report their ®ndings. However, it also tion capabilities downstream, but they need to be
has severe limitations. The general structure is able able to communicate with a large number of local
to implement the simpli®ed structure. entities. Agents may provide both data (in the
Of the tasks in the algorithms, only the lines form of status updates) and commands (for ex-
shown in italics are speci®c to each agent. All the ample, to request an additional module needed for
other functions are provided by the infrastructure execution), and the transceiver may need to send
to allow the creation of new agents in a minimum commands to the agents (for example, to set a
amount of time. parameter value). Transceivers also need to be able
to respond to commands from monitors, as well as
Algorithm 2. Polling structure for an agent. Lines provide them with status updates. Finally, if a
in italics represent sections that have to be pro- transceiver receives a message from an agent that it
vided by the author of the agent. is not able to interpret, the message should be
{Instantiation of the agent} forwarded to the transceiver's parent entity for
Instantiation-time initialization further processing.
{Execution of the agent}
Run-time initialization 3.3.2.5. Monitors. Monitors are the most complex
loop entities with respect to communication capabilities.
Perform checks A monitor must have all the functionality of a
if [abnormal condition detected] then transceiver, because it also needs to be able to
Generate STATUS_UPDATE message control local entities. Additionally it must be able
with new status information to start and control remote entities. In particular, it
if [STOP message was received] then must be able to start new transceivers or monitors
Cleanup and exit in remote hosts and communicate with them. The
Process other inputs monitor only communicates with monitors and
Sleep for a certain amount of time (inter- transceivers and not with all the agents that may be
check period) running in a host. This abstraction corresponds
with the AAFID architecture and communication
model, and it helps scalability because it reduces the
3.3.2.3. Filters. Filters are the only entities in the number of entities that a monitor has to track.
AAFID architecture that can communicate with A monitor must also be able to listen for con-
another entity at their same level in the hierarchy nections from remote entities. This is because a
E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570 559

transceiver or monitor may be started on its own Subclasses inherit all the base functionality
in a host and need to be controlled by an already from their subclass, and can extend it. For exam-
existing monitor. The new entity must be able to ple, a Monitor behaves essentially as a Control-
contact the monitor and register with it by sending lerEntity, but it adds functionality and modi®es
a CONNECT message. some of the features of the base class.
Once a remote entity is started or a connection
request has been processed, the monitor must be 3.3.4. Entity status
able to communicate with it in the same way as Each entity in AAFID2 is able to keep a status
with local entities. value, which represents whether it has detected a
Finally, monitors have the responsibility of problem, and the seriousness of the problem. Each
acting as repositories for all the information that entity must be able to report its status upon re-
their sub-entities may need. For example, a quest. The status of the whole intrusion detection
transceiver may be started with few local resourc- system is computed from the status of all the
es, and when it needs a module whose code is not individual entities.
locally present, it will request it of its monitor. The The status is kept using entity parameters. All
monitor must be able to locate the necessary code entities must keep its status as a numeric indicator
and provide it to the transceiver. If the monitor in a parameter called Status, and a textual de-
itself does not have the requested code, it must scription of it in a parameter called Message.
forward the request to its own controller entity. The value of Status must be a number between
If the code eventually arrives, the original re- 0 and 10 inclusive, where 0 means ``all normal''
quest from the transceiver must be immediately and 10 means ``extremely alarmed'', and the values
ful®lled. in between represent di€erent degrees of impor-
tance of the problem detected. No formal de®ni-
3.3.3. Object model tions have been given for the values of the status
Based on the requirements and design decisions indicator, and in AAFID2 it is up to each entity to
described in Section 3.3.2 we designed the class assign them.
hierarchy shown in Fig. 3. These classes corre-
spond almost directly with the functionality 3.3.5. Messages and commands
described in Section 3.3.2. In the Perl implemen- Messages in the communication model ¯ow
tation, all the class names are pre®xed with AA- through the edges of the graph shown in Fig. 2.
FID:: (for example, the full name of Entity is At the architectural level, we do not worry about
AAFID::Entity), but for the sake of space and the semantic contents of the messages, so we
clarity we only make reference to them by their de®ned a message format with the following ®elds
distinctive name in the rest of this paper. for AAFID2 messages:

Fig. 3. Class hierarchy in AAFID2 . Solid lines represent inheritance relationships between the object classes. Dashed lines represent
``uses'' relationships, meaning that a class uses another one internally. Grey ovals represent the classes in the main class hierarchy, that
implement the primary entity types. Dashed ovals represent auxiliary classes that are not instantiated as objects, but that contain
methods that are used by all the other classes.
560 E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570

· Message type. 3.4.1. Selection of a programming language


· Message subtype. AAFID2 is implemented in Perl 5 [28,33]. The
· Source identi®er. choice of this programming language was in¯u-
· Destination identi®er. enced by the following factors:
· Time stamp. · Ease of prototyping. Perl is a good language for
· Data. quickly prototyping and testing ideas. This was
This generic structure can be used by entities to one of our objectives for the AAFID2 proto-
represent any type of information. The kind of type, so Perl was a good choice.
message is denoted by the type ®eld. This ®eld also · Portability. Perl can be used in most versions of
implicitly determines the format of the data ®eld Unix, as well as on Windows 95 and NT. This
(the receiving entity must know the message type makes it extremely attractive to program a dis-
and must know how to interpret the data ®eld tributed application that we eventually want to
accordingly). The subtype ®eld can be used to run in a large number of systems.
provide additional information about the contents · Extreme flexibility. Being an interpreted lan-
of the message, as well as to indicate the speci®c guage, Perl provides an unprecedented ¯exibility
semantic interpretation that should be given to the in what a program can do. For example, we can
data ®eld. add code to a module at run time, even code
The source and destination identi®ers must provided by the user. This allows the modi®ca-
contain information that uniquely identi®es tion of parameters, internal variables, and even
the sending and receiving entities. The destina- subroutines without the need to restart the pro-
tion ®eld may be empty if the message is grams. In our prototype, it also makes it easier
sent over a one-to-one communication channel, to distribute code over the network, because
where there is no ambiguity about who the the code can be shipped as source and evaluated
receiver is. by the recipient.
The time stamp contains a unique representa- AAFID2 was implemented using the object-
tion of the time when the message was generated. oriented features of Perl 5. This makes it possible
AAFID2 uses the number of seconds elapsed since to directly implement the object model described
the Unix epoch (1 January 1970), which is a in Section 3.3.3.
commonly available value in programming lan- During the development and testing of AAFID2
guages under Unix. we came to also realize some of the disadvantages
Table 2 describes the base message types that of using Perl for the implementation:
we have de®ned for AAFID2 . The content of the · Big resource usage. The Perl interpreter is a large
data ®eld is speci®ed for each one. program, and it has to be in memory whenever a
The COMMAND message type provides en- Perl program is executed. This results in large
tities with the capability of de®ning their own memory and CPU usage footprints.
functionality, which can be accessed through · Excessive flexibility. The ¯exibility that we
special commands by parent entities or users. praised as an advantage can also be a problem.
We have also de®ned a basic set of commands For example, the lack of strict type checking and
to which any entity in AAFID2 must respond. the looseness with which Perl code can be writ-
This standard set of commands is shown in ten can lead to programs that are dicult to
Table 3. maintain when their size grows. We found that
we had to be particularly careful in documenting
and structuring the code to avoid problems.
3.4. Implementation notes Additionally, the object model in Perl has signif-
icant di€erences with those of other object-
We now describe some of the implementation oriented programming languages, which can
decisions and issues we faced while developing lead to confusions and awkwardness in the
AAFID2 . programs.
E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570 561

Table 2
Standard message types de®ned in AAFID2
NOTYPE
Subtypes N/A
Description Indicates that the message has not been initialized. This value can also be used in the message subtype
®eld if that ®eld has no speci®c value, or its value is irrelevant
Data ®eld N/A
CONNECT
Subtypes CHILD, PARENT, FILTER, GUI
Description Sent by an entity when it starts. It can be sent upstream to a parent entity, downstream to children
entities, and by ®lters and user interfaces. The subtype ®eld indicates who is sending the message
Data ®eld Information about the entity. In AAFID2 it contains the entity's identi®er and description
DISCONNECT
Subtypes CHILD, PARENT, FILTER GUI
Description Signals termination of the entity that sends the message. The purpose of this message is to allow the
receiving entity to perform actions to maintain consistent internal information
Data ®eld Same as in CONNECT
STATUS_UPDATE
Subtypes Irrelevant
Description The message contains a status update that the sender wants the recipient to have
Data ®eld Current status and descriptive message in the form of two sub®elds called Status and Message that
contain the current numeric status and a descriptive message of the situation. The format of these ®elds
is the Perl syntax for hash speci®cation. For example Status ˆ >0, Message ˆ >``No problems''
COMMAND
Subtypes Command name or RESULT
Description Speci®es a command that the receiving entity must execute. The subtype ®eld contains the name of the
command to execute. If a command produces a result, it will be sent back to the entity that requested the
command in a message of type COMMAND and subtype RESULT. Each entity can de®ne its own
commands in addition to the standard set. See Table 3 for the list of standard commands de®ned in
AAFID2
Data ®eld Named parameters to the command, in Perl hash-speci®cation format (Key ˆ > Value, separated by
commas). In a RESULT message, it contains the result produced by the command (which should also be
in the form of named parameters) plus a special parameter called Command that contains the name of
the command that produced the result
STOP
Subtypes Irrelevant
Description Stops the execution of the entity, performing any necessary cleanup actions. This should usually involve
at least sending a DISCONNECT message to any parent and children entities with which the entity has
communication
Data ®eld Irrelevant

· Security problems. Some of Perl's features, like 3.4.2. Entities and parameters
the ability to modify the code at run time, can In Perl 5, objects are represented by blessed
also lead to security problems. However, the references [33]. A reference can point to any kind
current implementation of AAFID2 is intended of Perl variable, including scalars, arrays, strings
as a proof of concept and not as a production or hashes. A common technique in Perl Object-
system, so this was a secondary concern for Oriented programming is to represent objects
our purposes. with a reference to an anonymous hash [5].
We believe that the most signi®cant advantages of Hashes in Perl are arrays where the indices can be
using Perl are the ease of prototyping and the ca- any value, not only numbers. In particular, the
pability to modify the code at run time. indices of a hash can be strings, and can be used
562 E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570

Table 3
Standard commands de®ned in AAFID2 a
STOP
Parameters None
Description Has the same e€ect as a STOP message
Returns N/A
EVAL
Parameters Code ˆ > ``Perl code to execute''
Description Allows the execution of arbitrary code in the context of the entity that receives the message. This
command is mostly for experimental purposes, for two reasons: it is easy to implement in Perl, but
may be impossible to achieve in other languages; and it opens the possibility for security
problems. The Code parameter contains the code to execute
Returns If the code executes without problems, does not produce a return value. If an error occurs, the
result contains an ErrorMessage parameter that contains the description of the error

SET_PARAMS
Parameters Parameter ˆ > Value pairs, separated by commas
Description Allows the speci®cation of values for internal entity parameters, as described in Section 3.4.2
Returns If the list of parameters and values contains an error, the command returns a description of the
error. Otherwise it produces no return value

GET_PARAMS
Parameters Params ˆ > ``param1, param2,. . .''
Description Allows the retrieval of internal entity parameters. The Params parameter must be a string
containing a comma-separated list of parameter names
Returns A list of parameter name±value pairs containing the requested parameters and their values
DUMP_YOURSELF
Parameters None
Description Instructs the entity to produce an internal representation of its current state
Returns A string representation of the entity (which is actually the current values of all its parameters),
contained in the Me parameter. This representation can be used to examine the internal state of
the entity, or possibly to initialize another entity to the same state
a
These commands are speci®ed as subtypes of a message of type COMMAND (see Table 2).

to store named values. An anonymous hash is a used as indices, and the value of each parameter is
hash that is not assigned to a variable, but that stored in the corresponding element of the hash.
can be accessed through a reference stored in a Because a single hash in Perl can store di€erent
variable. In our case, the reference to the anon- types of elements simultaneously, each parameter
ymous hash is stored as the representation of the can be of a di€erent type without causing any
object. problems.
Using an anonymous hash reference to repre- An example may make this mechanism clearer.
sent an object has the advantage that the hash can An agent may have the internal representation
be used as the object's internal name space. This shown in Fig. 4.
makes up for the lack of data inheritance in Perl's The notation Key ˆ > Value is used in Perl to
object model. Within its own hash reference (the specify the elements of a hash. Thus, we may de-
hash reference represents the object, so we nor- duct from the information in the ®gure that the
mally say ``within itself'') the object can store any agent has a check period (CheckPeriod) of 10 sec-
kind of values by name, as if they were instance onds and its current status (Status) is zero, among
variables. other things.
In AAFID2 each entity object is represented by Because each class instance has its own hash,
an anonymous hash that contains the parameters each entity can keep its own state separate from
for that speci®c entity. The parameter names are others.
E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570 563

3.4.3.2. Reacting to messages. The behavior of the


di€erent message types and commands is imple-
mented through regular Perl subroutines following
certain conventions, and thus is easy for someone
implementing a new entity type to add function-
ality.
To add support for a new message type (for
example NEWTYPE) a subroutine called mes-
sage_NEWTYPE (where the type name is in up-
percase) has to be implemented. When a message
Fig. 4. Sample internal representation of an agent.
of the new type is received, the subroutine will be
automatically invoked as an instance method of
The Entity class de®nes a number of methods the entity, which means that a reference to the
that can be used to set and query entity parame- entity will be its ®rst argument. The second argu-
ters. ment will be a Message object containing the
By convention, none of the parameters used message that triggered the call. The subroutine is
internally by the base classes in AAFID2 starts free to do any processing that it needs. If no return
with the pre®x ``My'', so authors of agents and value is necessary, the function should return the
other entities can use paramaters starting with that undef value. If a value is returned, it should be
pre®x (for example, MyUsers and MyKeys) assured another Message object, and it will be transmitted
that they will not have con¯icts with internal as-is to the entity that sent the original message.
parameters. We have provided a simpler interface for im-
plementing new commands, rather than new mes-
3.4.3. Messages sage types. For this reason we consider that most
3.4.3.1. Format. Messages in AAFID2 are repre- extensions to the basic infrastructure functionality
sented by objects of the class Message. The main should be done through new commands and not
purpose of this class is to store the ®elds as de®ned new message types. For each command there must
in Section 3.3.5, but also to allow its conversion to be a subroutine called command_CMD, where
and from a format suitable for sending over the CMD is the command name, in uppercase. When
network. Inside an entity the messages are stored the command is received, the subroutine will be
as an object, but before sending them to another called with three arguments: a reference to the
entity, they are converted to a single line of text entity itself, the Message object that triggered the
with the following format: call, and a hash containing all the parameter
name±value pairs passed in the data ®eld of the
TYPE SUBTYPE FROM TO TIME DATA
message. Although the whole message is available,
An entity that receives a message parses it and most commands should only need to examine the
converts it back into a Message object. Internally, hash to extract the values of their parameters and
all the ®elds are represented as strings. The DATA act accordingly. To produce a return value, the
portion of the string representation may contain command subroutine should return a hash refer-
spaces, depending on the contents of the DATA ence containing name±value pairs that will be sent
®eld. to the entity that requested the command in a
The utilization of strings to represent messages message of type COMMAND and subtype RE-
was decided to make it simpler for human users SULT. If there is no return value, the subroutine
to provide messages by hand and to monitor the must return undef.
¯ow of messages between entities. Future imple-
mentations may decide to use a more ecient 3.4.4. Communication mechanisms
format, particularly for sending messages over a The choice and implementation of the com-
network. munication mechanisms in AAFID2 was done
564 E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570

with two main objectives in mind: transparency causes the entity to block until the next time event
and system independence. To the extent that it is occurs.
possible, entities do not have to know whether Signals. Finally, the event mechanism can cause
they communicate over a network or within the an entity to react to Unix signals. The entity can
local host, even when completely di€erent mecha- de®ne the appropriate handlers, and they are
nisms may be used. We have put the portions of called automatically when the corresponding sig-
code that deal with the implementation di€erences nal occurs.
in speci®c places so that it is easy to locate and An entity can de®ne handlers for as many
modify. Most of the mechanism-speci®c or plat- events as it wants, including di€erent types of
form-speci®c code is in the Comm and Reactor events. The event engine, whenever possible,
classes. causes the entity to block until the next event
AAFID2 uses TCP connections for communi- occurs, to reduce resource usage.
cation over the network, and Unix pipes for
communication within the same host. These 3.4.6. Entity loading and execution
mechanisms were chosen because they are readily The ability to invoke a new local entity and
available in any Unix system, and both provide control it is implemented in the ControllerEntity
reliable end-to-end communication. The downside class, and thus is inherited by monitors and
is that none of them provide encryption or au- transceivers (see Fig. 3). The Monitor class extends
thentication capabilities. this capability by allowing the invocation of re-
mote entities. Additionally each entity must be
3.4.5. Event model able to run as a stand-alone program. We now
All entities in AAFID2 operate around a single describe the mechanisms used for each of these
event loop that processes di€erent types of events. types of entity execution modes.
Entities operate by de®ning the appropriate event
handlers and waiting for the corresponding events 3.4.6.1. Entity execution. Every entity must have a
to occur. The event mechanism is implemented by method called run that will be called as the entry
the Reactor class. point for the execution of the entity. When the
In the latest implementation of AAFID2 , an run method returns, the execution of the entity is
entity can set handlers for the following types of considered ®nished.
events: For instantiation-time initialization, every entity
File handles. In Unix systems most objects can may have a method called Init, which will be
be accessed through ®le handles. For AAFID2 , the called at the time the object is created. For run-time
most relevant objects are sockets and pipes. An initialization, an entity may de®ne a method called
entity can set a handler for a ®le handle, and the runtimeInit. The di€erence between the two
event loop will cause it to block until data are types of initialization can be seen in Fig. 5. In-
available. By blocking, the entity uses little re- stantiation-time initialization occurs when the new
sources while it is waiting. entity is created inside its corresponding trans-
Files. A common application for an agent or a ceiver, whereas run-time initialization occurs once
®lter is to read data from a ®le. A ®le is also ac- the entity starts executing in its own process.
cessed through a ®le handle. However, because of Therefore, any initialization that a€ects the state of
the way regular ®les are implemented in Unix, it is the process (for example, opening ®les, or creating
not possible for a process to block on ®le handle other child processes) has to be done in the run-
that represents a regular ®le. Therefore, the event time initialization to avoid a€ecting the transceiver.
engine processes regular ®les di€erently, by polling For cleanup activities when its execution ter-
periodically from them instead of blocking. minates, an entity may de®ne a method called
Time. The event engine can also react to time Cleanup.
events. Both one-shot and repeating events can be The Entity class provides the infrastructure for
scheduled by an entity, and the event mechanism a working entity, including placeholder methods
E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570 565

3.4.6.2. Stand-alone entity execution. To allow an


entity to be loaded as a stand-alone program, there
must be a mechanism for automatically creating
an instance of the class and invoking its run
method when the entity is executed. However, this
code must not be executed when the entity is
loaded from another one.
In AAFID2 , the mechanism used to solve this
problem is a subroutine called _EndOfEntity,
implemented by the Entity class, which must be
invoked in the last line in the source ®le of an
entity. This subroutine checks whether the ®le is
being loaded by itself or as a module from another
program. In the ®rst case, it creates an instance of
the entity, calls its run method, and exits when
execution terminates. In the second case, nothing
is done and a value of 1 is returned, which is the
customary way in Perl of signaling that a module
®le was loaded correctly.
When an entity is loaded stand-alone, its stan-
dard input and output are not redirected and the
user is able to type messages and see the results in
the terminal.
Fig. 5. Steps for loading and executing a local entity, called
newEntity in this example. Time is represented along the ver-
tical axis, and each vertical layer represents changes in the state 3.4.6.3. Local execution of an entity from another
of the entities as time passes. program. When an entity needs to be executed
from another program (for example, a transceiver
for run, Init, runtimeInit and Cleanup, as loading an agent), the following steps are per-
well as the code for automatically doing setup formed:
activities such as generating an entity identi®er, 1. Load the entity class using Perl's use state-
sending a CONNECT message up upon startup ment.
and a DISCONNECT message upon termination. 2. Create a new instance of the entity using its
These activities are done in the new method new method.
(constructor) de®ned by Entity, and should be 3. Create a new process, where the new entity will
inherited by all other entities. For this reason, be executed.
subclasses of Entity must not override the con- 4. Create two pipes between the parent and
structor, but provide their functionality through child processes, one for sending messages
Init and runtimeInit. from the parent to the child, and one for mes-
ControllerEntity and its subclasses recognize a sages from the child to the parent. In the
command called START, which receives a pa- child process, the reading end of one pipe is
rameter called Class that contains a speci®cation of associated with standard input and the writ-
the entity that needs to be started, in one of the ing end of the other one is associated with
following forms: standard output, therefore establishing the
· ``Classname'' requests the execution of a local ``up'' channel of the entity. In the parent pro-
entity of the given class. cess, the corresponding ends of the pipes are
· ``Host:Classname'' requests the execution of an stored in an internal parameter for future
entity of the given class in the host speci®ed. use, and to be able to receive messages from
Recognized only by Monitor. the new entity.
566 E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570

5. The child process executes the new entity by


invoking its run method. When it returns, the
process terminates.
These steps are illustrated in Fig. 5.

3.4.6.4. Remote execution of an entity. The Moni-


tor class provides the facilities for being able to
request the activation of an entity in a remote host.
The mechanism used by monitors to activate
remote entities is the following:
1. Check if a communication channel to a trans-
ceiver or a monitor in the requested host
already exists. If so, send to it a message to load
the required entity. Otherwise continue with the
following steps.
2. Execute in the remote host the Starter program,
with the appropriate parameters to tell it from
which host it was executed. AAFID2 uses ssh
(the Secure Shell) to execute Starter in the re- Fig. 6. Execution of a remote transceiver and entity using the
mote host. Starter program.
3. The monitor sets the appropriate ¯ags to indi-
cate that it is waiting for a connection from
the host where the Starter was executed, and The monitor knows about the network connec-
continues its normal execution. tion, because it receives the connection request
4. In the remote host the Starter instantiates a from the Starter in its server port.
transceiver (class PlainTransceiver), then con- · Once a network connection is established be-
tacts the server port in the monitor, establishes tween the monitor and the transceiver, it is kept
a TCP connection, and redirects its standard open. Future requests to start entities in the
input and output to it. same host will stop at the ®rst step of the algo-
5. The Starter runs the transceiver, which will then rithm described before, and simply cause the
communicate with the monitor (through its appropriate START message to be sent.
standard input and output, as redirected in
the previous step) to register with it. 3.4.6.5. Requests for missing code. When a con-
6. When the monitor receives the CONNECT troller entity receives a request to start a new en-
message from the newly created transceiver, it tity, it may be that some or all the code necessary
identi®es it as the one in which an entity has to load that entity (for example, its class source
to be started, and sends the appropriate com- ®le, or a module needed by it) is not present in the
mand to it. local host. In this case, the controller entity sends a
These steps are shown in Fig. 6. It is important to NEEDMODULE message up, specifying the
notice the following characteristics of this process: name of the missing class. The entity above must
· All entities are started locally (i.e., the transceiv- ®nd the appropriate code (probably by sending a
er is started locally by the starter, and the new NEEDMODULE message itself), and then send it
entity is started locally by the transceiver). It is down in a NEWMODULE message.
a separate program called Starter which estab- Once the controller entity receives the NEW-
lishes the network connection and does the MODULE message containing the code it needs, it
appropriate redirection of handles. For this saves it to a local ®le and tries to load it. If there is
reason the network connection is transparent still missing code (for example, the code received
to both the new transceiver and the new entity. needs another module that is not present), then the
E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570 567

process is repeated until an entity of the appro- (the monitor having to process the information
priate class can be instantiated. coming from a large number of transceivers). We
This mechanism allows the initial installation of have not experienced any of these situations be-
AAFID2 in the monitored systems to be limited to cause the AAFID2 implementation has not been
the essential classes, and then have all the others tested on a large network, but they could
(such as agents) deployed as needed. conceivably occur. The AAFID architecture,
however, allows for hierarchical deployment of
3.5. Experiences and future development monitors. By arranging monitors in a suitable
hierarchical fashion, most scalability problems
The AAFID2 prototype has several limitations, can be reduced by not allowing each monitor to
and through its testing, we have identi®ed some of receive more information than that which it can
them, and obtained other experiences. The main process without overloading the system in which it
ones we have identi®ed are in the following areas: is running.
Data analysis. No standard mechanism exists Rigidity of the architecture. We have identi®ed
for implementing data analysis in the transceivers situations where it may be useful to allow more
or in the monitors. The implementation of data ¯exible communication between entities in the
analysis mechanisms is complicated by the fact AAFID architecture, for example to allow one
that data coming from a single agent may need to agent to communicate directly with another agent.
be analyzed in several di€erent ways to look for This might also help avoid the scalability problems
di€erent problems. Therefore, it is unclear how the mentioned before.
messages from the agents have to be processed. None of these drawbacks invalidates the ap-
Impact. One of the undesirable consequences of proach used by the AAFID architecture, because
implementing AAFID2 in Perl is that it has large they can be addressed by engineering e€ort or
resource needs. In an experiment running nine further research, and because AAFID can be used
entities (counting agents and ®lters) on a Sparc as a platform for performing the necessary re-
Ultra 1 with 128 MB of RAM, each entity con- search.
sumed an average of 2381.9 kB of memory, or To address these issues and some others, we
1.8%. This may not seem exceedingly expensive, have determined some points for future work in
but for AAFID to be useful, we need to have a the AAFID2 prototype:
large number of agents running on each host, Reduction modules. We have designed a new
certainly in the tens and possibly in the hundreds. entity for the AAFID architecture called reduction
Complexity. In some cases, the AAFID2 code modules, which is intended to provide ¯exible data
has grown more complex than apparently neces- analysis capabilities. A reduction module will be
sary. For example, in our testing, we have not used designed to detect certain intrusions, and will
the automatic distribution of code that is possible consist of a list of agents needed, plus analysis
with AAFID2 , because the whole prototype has code to execute on the transceivers and the moni-
been available to all the machines in the network. tors. This allows the user to specify the capabilities
Having this feature makes the code more complex required from the intrusion detection system.
and dicult to maintain. Use of threads in Perl. In the current AAFID2
Scalability. Although the distribution of tasks prototype, agents and ®lters are implemented as
between the di€erent AAFID entities aids scala- separate processes. We are considering making
bility, there is still the possibility of bottlenecks at each entity a thread within the corresponding
the controller entities (particularly monitors, but transceiver. This will reduce the resource usage
also transceivers) when there are a large number and the impact of AAFID2 on the systems where it
of hosts being monitored, and a large number of runs.
agents in each host. The bottleneck can be both in Porting AAFID2 to other architectures. Work is
terms of communication (many entities sending underway to port AAFID2 to Windows NT.
messages to the same monitor) and processing The porting e€ort has provided insight into the
568 E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570

inherent incompatibilities between Windows NT types of data may need to be collected at di€er-
and Unix, but progress has been made, and being ent points.
able to run AAFID2 under NT will allow us to · How do we communicate the requirements and
perform even more experimentation with di€erent the data between the collection and the analysis
intrusion detection techniques. mechanisms?
Embedded sensors. We have also started work · How do we eciently collect data without dis-
on building sensors embedded into the code of the turbing system operation?
operating system and its programs. This will make · How do we represent and store the data?
it possible to obtain information at the point Data analysis. Once we have the data, they must
where it is generated, and will also enable the be analyzed to detect intrusions. In this area we
creation of sensors that are lighter and more re- have the following questions:
sistant. · Where do we analyze it? In a central location?
Advanced architecture capabilities. Some of the At the point of collection?
capabilities o€ered by the AAFID architecture (for · How do we analyze it? Di€erent detection re-
example, arranging monitors in a hierarchical quirements may need di€erent types of analysis.
fashion) are not fully implemented in the latest However, we do not know how to determine
versions of AAFID2 . We plan on fully imple- which analysis techniques are better suited for
menting the architecture in future versions of the detecting di€erent types of problems or for ana-
prototype. lyzing di€erent types of data.
· How do we measure ``better'' in the previous
question? It is not clear which criteria can be
4. Research directions used to compare analysis techniques.
Reaction. Assuming we have successfully ana-
The main objective of the AAFID architecture lyzed the data, the question remains of how to
and its prototype is to serve as a platform for re- react when problems are detected. And this area
search in innovative intrusion detection tech- includes the following three points:
niques. As such, it has helped us identify questions · How do we present the results? Ultimately, a hu-
that have not been completely answered by past man user must be able to see the results of the
intrusion detection research. We classify these analysis and take a decision. But if the intrusion
questions in the following areas: detection system is not capable of presenting its
Detection and data requirements. The ®rst step results in a form that is understandable to the
in the design of an intrusion detection system is to user, then its whole purpose is defeated.
determine what it needs to detect. Research ques- · Which control structure to use? In a distributed
tions include: intrusion detection system, some components
· How do we express what we want to detect? (de- control others, and the user must be able to ex-
termine the detection requirements). ert control to react to detected problems. Re-
· From the detection requirements, how do we de- search is needed to determine which control
termine which data are needed? (determine the structures are best for di€erent situations.
data needs). · Should the intrusion detection system react
· How do we express the data needs? automatically? It is possible for the intrusion
· How do we verify that the data needs are su- detection system to react automatically to
cient to satisfy the detection requirements? certain problems to try to contain or stop the
Data collection. Once we know what data are damage. However, automatic reaction creates
needed, the next step is to collect it. In this area, the possibility of further problems when false
research questions include: positives occur. Further research is needed to
· Where and how to collect the data? Data may be determine under which situations it is safe to
collected at di€erent points in the system (e.g., react automatically, and which types of reaction
application level or kernel level), and di€erent are appropriate.
E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570 569

These questions are only some of the most [12] D.E. Denning, P.G. Neumann, Requirements and model
relevant open research areas in intrusion detection. for IDES ± a real-time instrusion detection system,
Technical Report, Computer Science Laboratory, SRI
It is a young ®eld, and we are only starting to International, August 1985.
understand some of the problems involved. [13] D. Farmer, W. Venema, Computer forensics analysis class
handouts. Web page at http://www.fish.com/
forensics/, accessed in May 2000, August 1999.
[14] W.M. Farmer, J.D. Guttman, V. Swarup, Security for
References mobile agents: issues and requirements, in: Proceedings of
the 19th National Information Sytems Security Confer-
[1] J.S. Balasubramaniyan, J.O. Garcia-Fernandez, E. Spaf- ence, vol. 2, National Institute of Standards and Technol-
ford, D. Zamboni, An architecture for intrusion detection ogy, October 1996.
using autonomous agents, Technical Report 98-05, [15] R. Heady, G. Luger, A. Maccabe, M. Servilla, The
COAST Laboratory, Purdue University, May 1998. architecture of a network level intrusion detection system,
[2] K.A. Bradley, S. Cheung, N. Puketza, B. Mukherjee, R.A. Technical Report, University of New Mexico, Department
Olsson, Detecting disruptive routers: a distributed network of Computer Science, August 1990.
monitoring approach, in: Proceedings of the 1998 IEEE [16] L. Heberlein, G. Dias, K. Levitt, B. Mukherjee, J. Wood,
Symposium on Security and Privacy, May 1998. D. Wolber, A network security monitor, in: Proceedings of
[3] J.M. Bradshaw, An introduction to software agents, in: the IEEE Symposium on Research in Security and Privacy,
J.M. Bradshaw (Ed.), Software Agents, AAA1 Press/MIT May 1990.
Press, Cambridge, MA, 1997, pp. 3±46 (Chapter 1). [17] J. Hochberg, K. Jackson, C. Stallings, J.F. McClary,
[4] S. Cheung, R. Crawford, M. Dilger, J. Frank, J. Hoagland, D. DuBois, J. Ford, NADIR: an automated system for
K. Levitt, J. Rowe, S. Staniford-Chen, R. Yip, D. Zerkle, detecting network intrusion and misuse, Computers and
The design of GrIDS: a graph-based intrusion detection Security 12 (3) (1993) 235±248.
system, Technical Report CSE-99-2, Department of Com- [18] S.A. Hofmeyr, An immunological model of distributed
puter Science, University of California at Davis, Davis, detection and its application to computer security, Ph.D.
CA, January 1999. thesis, University of New Mexico, May 1999.
[5] T. Christiansen, Tom's object-oriented tutorial for Perl [19] IEEE, IEEE Journal on Selected Areas in Communica-
manual page included with the Perl 5 distribution, April tions (Special issue on Secure Communications), May
1998. 1989.
[6] Cisco Systems, Cisco netranger. Web page at http:// [20] G.H. Kim, E.H. Spa€ord, The design and implementataion
www.wheelgroup.com/univercd/cc/td/doc/pcat/ of Tripwire: a ®le system integrity checker, in: J. Stern
nerg.htm, accessed in May 2000. (Ed.), The Second ACM Conference on Computer and
[7] M. Crosbie, B. Dole, T. Ellis, I. Irsul, E. Spa€ord, IDIOT ± Communications Security, ACM Press, Fairfax, VA,
Users Guide, COAST Laboratory, Purdue University, 1398 November 1994.
Computer Science Building, West Lafayette, IN 47907- [21] S. Kumar, Classi®cation and detection of computer intru-
1398. ftp://coast.cs.purdue.edu/pub/COAST/ sions, Ph.D. Thesis, Purdue University, West Lafayette, IN
papers/IDIOT/IDIOT_Users_Guide.ps, September 47907, 1995.
1996. [22] T.F. Lunt, R. Jagannathan, R. Lee, S. Listgarten, D.L.
[8] M. Crosbie, E. Spa€ord, Defending a computer system Edwards, P.G. Neumann, H.S. Javitz, A. Valdes, Devel-
using autonomous agents, Technical Report 95-022, opment and application of IDES: a real-time intrusion
COAST Laboratory, Department of Computer Sciences, detection expert system, Technical Report, SRI Interna-
Purdue University, West Lafayette, IN 47907-1398, March tional, 1988.
1994. [23] T.F. Lunt, A. Tamaru, F. Gilham, R. Jagannathan, P.G.
[9] M. Crosbie, E. Spa€ord, Defending a computer system Neumann, H.S. Javitz, A. Valdes, T.D. Garvey, A real-
using autonomous agents, in: Proceedings of the 18th time intrusion detection expert system (IDES) ± Final
National Information Systems Security Conference, Octo- Technical Report, Technical Report, SRI Computer Sci-
ber 1995. ence Laboratory, SRI International, Melno Park, CA,
[10] M. Crosbie, G. Spa€ord, Active defense of a computer February 1992.
system using autonomous agents, Technical Report 95-008, [24] Merriam-Webster, Intrusion Merriam-Webster OnLine:
COAST Group, Department of Computer Sciences, Pur- WWWebster Dictionary. http://www.m-w.com/
due University, West Lafayette, IN 47907-1398, February dictionary, accessed on 16 May 1998.
1995. [25] B. Mukherjee, T.L. Heberlein, K.N. Levitt, Network
[11] D.E. Denning, D.L. Edwards, R. Jagannathan, T.F. Lunt, intrusion detection, IEEE Network 8 (3) (1994) 26±41.
P.G. Neumann, A prototype IDES ± a real-time intrusion [26] P.A. Porras, P.G. Neumann, EMERALD: Event monitor-
detection expert system, Technical Report, Computer ing enabling responses to anomalous live disturbances, in:
Science Laboratory, SRI International, 1987. Proceedings of the 20th National Information Systems
570 E.H. Spa€ord, D. Zamboni / Computer Networks 34 (2000) 547±570

Security Conference, National Institute of Standards and Eugene H. Spa€ord is a professor of


Technology, 1997. Computer Sciences at Purdue Univer-
[27] T.H. Ptacek, T.N. Newsham, Insertion, evasion, and denial sity, the University's Information Sys-
tems Security Ocer, and is Director
of service: eluding network intrusion detection, Technical of the Center for Education Research
Report, Secure Networks, January 1998. Information Assurance and Security.
[28] E. Siever, D. Futato, Perl Module Reference, vol. 1&2, CERIAS is a campus-wide multi-
O'Reilly, Sebastopol, CA, included in the Perl Resource disciplinary Center, with a broadly-
focused mission to explore issues
Kit, November 1997. related to protecting information and
[29] S.R. Snapp, J. Brentano, G.V. Dias, T.L. Goan, L.T. information resources. Spaf has
Heberlein, C. Lin Ho, K.N. Levitt, B. Mukherjee, S.E. written extensively about information
Smaha, T. Grance, D.M. Teal, D. Mansur, DIDS (distrib- security, software engineering, and
professional ethics. He has published
uted intrusion detection system) ± motivation, architecture, over 100 articles and reports on his research, has written or
and an early prototype, in: Proceedings of the 14th contributed to over a dozen books, and he serves on the
National Computer Security Conference, Washington, editorial boards of most major infosec-related journals.
DC, October 1991. Dr. Spa€ord is a Fellow of the ACM, Fellow of the AAAS,
senior member of the IEEE, and is a charter recipient of the
[30] S.R. Snapp, S. Smaha, D.M. Teal, T. Grance, The DIDS Computer Society's Golden Core award. Among other activi-
(distributed intrusion detection system) prototype, in: ties, he is chair of the ACMs US Public Policy Committee, a
Proceedings of the USENIX Summer 1992 Technical member of the Board of Directors of the Computing Research
Conference, San Antonio, TX, June 1992. Association, and is a member of the US Air Force Scienti®c
Advisory Board. He regularly serves as a consultant on
[31] S. Staniford-Chen, S. Cheung, R. Crawford, M. Dilger, information security and computer crime to law ®rms, major
J. Frank, J. Hoagland, K. Levitt, C. Wee, R. Yip, D. corporations, US government agencies, and state and national
Zerkle, GrIDS: a graph based intrusion detection system law enforcement agencies around the world. More information
for large networks, in: Proceedings of the 19th National may be found at http://www.cerias.purdue.edu/homes/spaf. In
his spare time, Spaf wonders why he has no spare time.
Information Systems Security Conference, vol. 1, Na-
tional Institute of Standards and Technology, October
1996. Diego Zamboni is a Ph.D. student at
[32] K.M.C. Tan, D. Thompson, A.B. Ruighaver, Intrusion Purdue University, where he is work-
ing in CERIAS in Intrusion Detection
detection systems and a view to its forensic applications, research. He obtained his M.S. in
Technical Report, Department of Computer Science, Computer Science from Purdue Uni-
University of Melbourne, Parkville 3052, Australia. versity. Previously he obtained his
http://www.securityfocus.com/data/library/ bachelor's degree in Computer Engi-
neering from the National Autono-
idsforensics.ps. mous University of Mexico, where he
[33] L. Wall, T. Christiansen, R.L. Schwartz, Programming was in charge of the security for the
Perl, 2nd ed., O'Reilly, Sebastopol, CA, September 1996. Unix machines at the Supercomputing
[34] G.B. White, E.A. Fisch, U.W. Pooch, Cooperating security Department. He also established the
University's Computer Security Area,
managers: a peer-based intrusion detection system, IEEE one of the ®rst Computer Security
Network (1996) 20±23. Incident Response Teams in Mexico.