This article has been accepted for publication in Computer but has not yet been fully edited.

Some content may change prior to final publication.

An Empirical Study of Commercial Antivirus Software Effectiveness

Orathai Sukwong Hyong S. Kim James C. Hoe
Electrical and Computer Engineering
Carnegie Mellon University, Pittsburgh, PA
{osukwong,kim,jhoe}@ece.cmu.edu

Abstract
Despite the widespread use of antivirus software (AV) malware is pervasive in today’s computing environment.
This paper presents an empirical study on the effectiveness of six commercial AVs against Windows malware
collected from April to July 2009. A subset of the studied AVs performs new-to-the-market behavior-based
detection in addition to traditional signature-based detection. The study shows that the AVs can identify at most
62.15% of the malware on the first day of collection. 8.61% of the malware are not detected by any of the studied
AVs for more than a month after being collected. During the malware’s execution, the AVs with behavior-based
detection provide protection against modifications to certain system and network registry keys, but leave some
keys (e.g. a Microsoft-Word key controlling macro execution) unprotected, leading to security breaches. Behavior-
based detection also prevents code injection and malicious behaviors (e.g. download non-executable files with
executable content), but not all of them.

Keywords: Operating systems. Security and privacy protection. Invasive software.

1 Current State of Antivirus Software

This paper reports the effectiveness of the protection offered by current AVs against contemporary malware
currently in circulation. The study quantifies the fraction of malware detected successfully by the AVs during the
study period and the time required for detection after the malware’s initial appearance. The study also records the
AVs’ responses to malware’s execution.

1.1 Antivirus Detection Mechanism

Users rely on antivirus software (AV) to keep their computers free of malware. We refer malware to any malicious
software which is detected by one or more AVs. AV products employ two common detection mechanisms:
signature-based detection and behavior-based detection.

1.1.1 Signature-based Detection

Signature-based detection is a static method most commonly used by AVs. As illustrated in Figure 1, when a
computer receives a new file, the signature-based detector scans the file. If the file contains a byte sequence that
matches one of the known threats' byte-based identifications, the file is considered a risk and is quarantined.
These byte-based identifications are commonly known as signatures. Traditional signatures are typically derived by
analyzing the contents of files that have been confirmed to be malicious. This file analysis takes time. Traditional
signatures rarely cause false-positive detections (i.e. identifying benign software as malware), but cannot detect
new, not-previously-identified malware for which no signature is available.

Besides traditional signatures, some AVs also statically look for program-logic signatures that identify specific
behaviors in an executable file. For example (from Symantec White Paper Series Volume XXXIV), if a program

Digital Object Indentifier 10.1109/MC.2010.187 0018-9162/$26.00 ) 2010 IEEE

 This article has been accepted for publication in Computer but has not yet been fully edited.
Some content may change prior to final publication.

matches the signature "B8 02 3 BA ?? ?? CD 21", it means the program opens a file. These program-logic signatures
are commonly referred to as heuristic signatures. Heuristic signatures can also exploit a unique feature of
malware. For example, AVs create packer identifiers and use them to heuristically classify whether a file is
malicious. A packer itself is neither malicious nor benign but it can be used to pack malicious or benign files. These
packer identifiers are derived from packers that are currently found in malware that packs only malicious files.
Heuristic signatures can potentially identify not-previously-identified malware, but can also lead to false positive
identification because benign programs could also match a flagged heuristic signature. In addition, malware may
also use uncommon or obfuscated implementations of flagged behaviors to evade detection using specific
heuristic signatures.

Traditional and heuristic signatures are not resilient to code obfuscation, such as variable renaming and garbage
insertion [1]. Code obfuscation techniques change the byte-level representation of the malware so that their byte
sequences do not match the signatures in an AV’s signature collection. A polymorphic virus is an example of an
obfuscated threat. It consists of three main components: decryption routine, mutation engine and virus body.
When it infects other programs, it creates a copy of itself and uses the mutation engine to create a new
randomized decryption routine in memory. It encrypts the mutation engine and its copy. This encrypted code and
the new decryption routine are then inserted into other programs. Hence, this new polymorphic virus (infected
file) has little or no resemblance to its parent, although their functions are the same. To detect this virus, Symantec
employs a CPU emulator to allow the virus to decrypt and reveal its malicious code without harming the system
(from Symantec’s paper on Understanding and Managing Polymorphic Viruses), shown in Figure 1 as the pre-
execution stage. Then, the AVs can proceed with a signature scan on the malicious code and mutation engine. CPU
emulation is a relatively slow process. The AVs have a very limited amount of time from the moment a user clicks
on a file to the time the file is uploaded to emulate an unknown program. Additionally, virus may intentionally
delay the decryption process or contain a logic trick (e.g. infect only at midnight). The AVs thus have to heuristically
rely on other characteristics to classify the file.

In signature-based detection, researchers have been focusing on improving signature accuracy as well as
developing an automated process for signature generation to combat with fast automatic attacks (e.g.
polymorphic worm [10]). The speed of signature scanning is also important due to a continuous growth of data.
The amount of time and computing resources required for scanning increases as the number of signatures
increases. For example, the approach proposed by [8] doubles scanning speed and reduces memory consumption.

1.1.2 Behavior-based Detection

Behavior-based detection dynamically examines the program’s execution behavior and then classifies the program
as malicious or benign based upon its behavior, which remains unchanged even after the code obfuscation.
Behavior-based detection has the potential to detect new malware by monitoring system activities, configuration
changes, network communications, and user interaction (see Figure 1). It can also provide swift protection against
dangerous executions by preventing actions that violate pre-defined execution restrictions. However, it can cause
unacceptable false positive rates because benign programs can behave similarly to malicious programs.

Behavior-based detection has been researched in academia for more than a decade and integrated into
commercial AV products in recent years. One area that researchers have been investigating is how to accurately
identify malicious behaviors with minimal false positives. Beginning with Forrest et al [2], they used deviations
from a program’s expected normal behaviors in terms of their sequences of system call as maliciousness
indicators. This approach is called anomaly detection. Later works [6, 7, 9] enhanced detection accuracy by
incorporating additional information, such as data flow and control flow, into normal behavior models. However,
anomaly detection does not work well with the rapid changes in software development. By definition, new

Digital Object Indentifier 10.1109/MC.2010.187 0018-9162/$26.00 ) 2010 IEEE

 This article has been accepted for publication in Computer but has not yet been fully edited.
Some content may change prior to final publication.

programs are unknown and, therefore, have no expected normal behavior. Many researchers [4, 5, 11] adopted
misuse detection, which instead classifies a program based on behaviors deemed malicious. Early behavior-based
detection suffers from expensive overhead [4, 11] and unacceptable false positive rates [3]. More recently
researchers have shown promising experimental results in terms of detection accuracy and run-time performance.
How these solutions will perform in uncontrolled and untested environments, especially in term of false positives,
is an open question.

Figure 1. Antivirus Operation.

1.2 Commercial Antivirus Evaluations

AV-Comparatives.org is a well-known AV testing organization. In December 2009, they release their first dynamic
test assessment which was performed manually by four people. Unlike their testing which lasted only ten days and
used 100 test cases, our study covers a five-month period and automatically examines 1,115 distinct malware
samples. Unlike AV-Comparatives.org which conducts testing every three months on data sets that are frozen
before testing, we test every malware file every day using the daily-updated AVs from the moment we download
the malware until it is quarantined. Our testing captures an AV protection capability at the moment the new
malware is present. AV-Test.org, another well-known AV testing organization, provides comparative detection
results for proactive detection including behavior-based detection and response times to outbreaks. These results
do not provide the details of which threats AVs missed or detected; they merely rank detection on a five-point
scale from very good to very poor. The testings by AV-Comparatives.org and Virusbtn.com also include evaluations
of the AVs’ false-positive rate against legitimate programs; our study does not evaluate the AV’s false-positive rate.

Digital Object Indentifier 10.1109/MC.2010.187 0018-9162/$26.00 ) 2010 IEEE

 This article has been accepted for publication in Computer but has not yet been fully edited.
Some content may change prior to final publication.

2 Methodology and Setup

2.1 Malware Collection

According to Symantec Threat Report (Volume XV), web-based attacks were the most preferred means of malware
infection. To gather malware for the study, we download 18,300 suspicious web pages identified by
stopbadware.org over four months (April – July, 2009). We focus on Windows-compatible malware files with EXE,
SYS and DLL suffixes. Files with these suffixes are likely to be malicious because they indicate file types not
normally found in web pages. We also include some document files (DOC, XLS and PDF). We select approximately
2,100 unique suspicious files from about 230,000 files downloaded. If a chosen file is later found to be benign, we
remove that file from the data set. We are ultimately left with 1,115 unique malicious files. About 80% of the
malware has an EXE suffix and 10% has a PDF suffix. The files also include various types of malware (e.g. Trojan,
worm, virus, spyware, rootkit, etc), but the majority of them (75%) are Trojans.

2.2 AVs
For each new malware we collected, we subject it to an assay of six well-known commercial AV scans:

x Avast! 4.8 Professional version 4.8.1335 (Avast)

x Kaspersky Internet Security 2009 (Kaspersky)
x McAfee Total Protection with Security Center version 9.15 (McAfee)
x Norton Internet Security 2009 version 16.5.0.135 (Norton)
x Symantec AntiVirus version 10.1.7.7000 (Symantec)
x Trend Micro Internet Security Pro version 17.1.1250 (Trend Micro)

We install the AVs with their default options, except for the automatic update feature. In order to control the
granularity of the virus-signature update, we perform the update on a daily basis. By default, Symantec updates
daily, while Avast, Kaspersky, McAfee, Norton, and Trend Micro update several times a day.

For each malware and for each AV, we repeat the scans every day using up-to-date signatures to determine how
many days it takes the AV to successfully detect the file after being collected. During a daily scan, if a malware file
passes an AV’s signature scan, we proceed to execute the malware file for 3~5 minutes to observe the AV’s
response by behavior-based detection. Among the six commercial AVs studied, Kaspersky, McAfee, Norton and
Trend Micro claim to have behavior-based detection.

In the experiment, we provide unrestricted Internet access. Certain malware requires Internet access to exhibit
their malicious behaviors (e.g. communicate with a remote server and download other malware). Internet access
also allows the AVs to reach their best detection capability. In the last few years, several vendors have brought in
other technology to enhance their AV. McAfee Corp. has a cloud-based service called Artemis, which allows clients
to access their latest file analysis database in the AV’s servers. This service helps reduce the delay in the virus-
signature-update distribution, which is typically released in batches. Kaspersky, Norton and Trend Micro also offer
similar services. Additionally, we enable an optional feature that anonymously and automatically submits security-
related information to the AVs’ servers. This feature may help reduce the AV’s detection time of unknown
malware. This feature is available in Kaspersky, McAfee, Norton, and Trend Micro.

2.3 Execution Environment

The experiments in the study are conducted using virtual machine (VM) environments in order to isolate and
contain the damage caused by malware’s execution. A fresh, uncorrupted VM with up-to-date signatures is used to

Digital Object Indentifier 10.1109/MC.2010.187 0018-9162/$26.00 ) 2010 IEEE

 This article has been accepted for publication in Computer but has not yet been fully edited.
Some content may change prior to final publication.

carry out the scan in each malware/AV combination each day to prevent other malware interference. Each VM
runs Window XP SP2 with Wireshark (wireshark.org) to monitor network traffic. Other applications, such as Adobe
Acrobat and Reader 9.1, Microsoft Office 2003, are installed in the VMs. Since we have to repeat the experiments
daily until a malware file is classified as a risk, we use a VM management program, VManager [12], to assist in
creating VMs on demand. We supply a base image for each selected AV (six VMs base images total) to VManager.
VManager creates a new VM with a copy-on-write disk image of the requested base image for every file
inspection. We thus ensure every VM to be intact and initially uncontaminated.

3 Study Results
In this section, we present the results of the five-month study (April-August, 2009) separately in two stages of AV
detection illustrated in Figure 1. The first stage (quarantine file) begins when the malware enters the system and
ends before its execution, as shown in Figure 1. In this stage, we quantify the effectiveness of the AVs in
preventing malware execution. The second stage, called block program behavior, occurs when the AVs do not
detect malware by signature-based scan and let it proceed to execute. In this stage, we record the AVs’ responses
and any consequences to the malware’s execution based on the log reported by the AVs. This experiment assumes
that there is no difference in malware execution between systems with and without AV installed.

3.1 The First Stage: Quarantine File

For each AV, Figure 2 groups the malware sample population by the numbers of days from the first day we
download malware to the day the AVs can detect it. The longer it takes for the AVs to identify a malware program,
the greater the chance for it to cause damage.

100
> 30 days
90
80 17-30 Days
70 9-16 Days
% of Malware

60
5-8 Days
50
3-4 Days
40
30 2 Days
20 1 Day
10
Zero Day
0

Figure 2. Percentage of the malware detected by the AVs at different number of days from zero days.

According to the results, the AVs immediately detect up to 62.15% of the malware and require days or even weeks
to detect the rest. As shown in Figure 2, 4.21%-16.50% of the malware are detected between 8 and 30 days, and
th
8.52%-21.88% of the malware are not detected by the 30 day.

Digital Object Indentifier 10.1109/MC.2010.187 0018-9162/$26.00 ) 2010 IEEE

 This article has been accepted for publication in Computer but has not yet been fully edited.
Some content may change prior to final publication.

To understand how long malware can escape detection, we also scan 260 unique malware files collected from
November 11-17, 2008 using the AVs with the signature update of August 29, 2009. Even then, the AVs do not
detect all of the malware, despite all the malware being in circulation for more than 8 months. From this
collection, Avast, Kaspersky, McAfee, Norton, Symantec and Trend Micro detect 96.92%, 81.92%, 98.46%, 95.77%,
95.77%, and 91.54% of Malware2008 respectively.

We also observe that having behavior-based detection does not seem to improve the AVs’ ability to quarantine
malicious programs before they are executed. As shown in Figure 2, Avast (no behavior-based detection)
eliminates roughly the same number of known malware as Kaspersky, McAfee and Norton (with behavior-based
detection). We also compare Norton and Symantec (with and without behavior-based detection) which have the
same heuristic detection technology and presumably comparable signature-based detection because they are
from the same vendor. In Figure 2, Norton detects ~6% more known malware than Symantec. This discrepancy
comes from the difference in default heuristic detection level. By default, Norton sets a higher heuristic detection
level than Symantec – meaning that heuristic detection in Norton is more sensitive than in Symantec. For example,
Norton detects the malware W32.Virut.CF ten days before Symantec does because a heuristic signature that
indicates a morphing or encrypting routine (Suspicious.MH690.A) is triggered in Norton. Symantec could detect
this malware as well if we change its default heuristic detection setting to be the same as Norton’s.

Trojan-PSW.Win32.LdPinch.smt Trojan-PSW.Win32.LdPinch.gxh
Trojan-PSW.Win32.LdPinch

Trojan-GameThief.Win32.WOW

Trojan-
GameThief.Win32.OnLineGames
Trojan-Downloader.Win32.Injecter

Trojan-
Downloader.Win32.FraudLoad
Trojan.Win32.Qhost

Trojan.Win32.Buzus

Net-Worm.Win32.Koobface

Exploit.Win32.Pidief

Exploit.JS.Pdfka

Backdoor.Win32.Wuca

Backdoor.Win32.HareBot

Figure 3. Timeline of the unique malware variants in the malware collection in different families from the day
they were downloaded until they were detected by Kaspersky.

A significant shortcoming in signature-based detection is its resilience against malware variants. In Figure 3, each
row contains the unique malware variants from the same families in the malware collection. Each line represents
the timeline for each variant from the date they are downloaded until the date they are detected by Kaspersky. As
shown in Figure 3, although the AV initially detects some variants in the family, it cannot immediately detect all
new variants in the same family. The AV can take days or months to correctly identify unknown variants. For
example, Trojan-PSW.Win32.LdPinch.smt is downloaded on June 29, 2009 and is detected immediately, while

Digital Object Indentifier 10.1109/MC.2010.187 0018-9162/$26.00 ) 2010 IEEE

 This article has been accepted for publication in Computer but has not yet been fully edited.
Some content may change prior to final publication.

Trojan-PSW.Win32.LdPinch.gxh is downloaded on July 15, 2009 but not detected until August 2, 2009. The result
also shows that many malicious samples pose challenges to more than one AV. 87-94% of the malware that each
AV takes more than 30 days also take at least another AV more than 30 days to detect. Some of these malware
samples use encrypting or injecting-code techniques which require complex analysis to understand.

To understand the risk of malware, we use risk-level and damage-level information from the malware descriptions
provided by Symantec, McAfee and Trend Micro. Risk-level assessment is a combination of infection/spread
distribution and damage potential. All malware in the data set are rated as low risk. However, such threat may
pose medium or high damages, such as infecting other programs and stealing credentials. Several malicious files
that are detected after 30 days have medium and high damages. For example, 17%, 16.52%, and 0.87% of the
malware that Kaspersky takes more than 30 days to detect are rated as low, medium and high-level damage
respectively.

3.2 The Second Stage: Block Program Behavior

Malware undetected by signature scan is executed to observe the AV’s responses and the effects on the system
due to the execution. We focus on three major subsystems: file system, memory and network.

3.2.1 File System

We study how the AVs react when the malware attempts to create and modify the file system, including user files,
system files and Windows registry.

3.2.1.1 Files
Malware often leaves one or more persistent copies of its malicious code in the system so that it remains after the
system reboots. These copies are sometimes written to files with apparently benign names (e.g. install.exe,
AV.exe, etc), or inserted into other benign programs (file infection). They can also be downloaded from remote
hosts. We call these copies child files. We measure the number of undetected malware generating child files, how
many of them have malicious child files quarantined by the AVs, and what protection the AVs provide to keep
system files intact.

Our results suggest that after executing undetected malware, it is likely that the system will have additional
malware. As seen in Figure 4, 42.83% of the malware is undetected on the first day we collect it. Out of these
undetected malware files, 53.83% produces one or more malicious child files. We refer to these malware files as
undetected malware with malicious child files. 23.49% of the undetected malware files with malicious child files
have one or more child files that are detected within zero days by the AVs. 12.14% of the undetected malware files
with malicious child files have one or more child files that are detected more than zero days by the AVs. 64.36% of
the undetected malware with malicious child files has malicious child files that are not detected by the AVs. Note
that we consider child files with EXE, SYS and DLL extensions to be suspicious. The statistics reported here are the
average values of all six AVs.

To mitigate file-system intrusions, the AVs with behavior-based detection impose restrictions on certain system-
wide changes (e.g. modifying system or library (DLL) files). System and library files are an attractive target because
they are frequently loaded and necessary for other programs to run properly. From the experiment, Kaspersky
prevents three malicious programs from creating/modifying EXE and DLL files in system folders. McAfee blocks 11
malicious programs from automatically starting at system boot-time by adding themselves to the startup folder.
However, malware can increase its probability of execution in the system by attaching itself to a wide range of
files. We find that AVs typically allow malware to stay in temporary and program folders (with benign or system-

Digital Object Indentifier 10.1109/MC.2010.187 0018-9162/$26.00 ) 2010 IEEE

 This article has been accepted for publication in Computer but has not yet been fully edited.
Some content may change prior to final publication.

file names). Additionally, Kaspersky, McAfee and Trend Micro block changes to the hosts file (containing the
mapping between IP addresses and hostnames), while Norton allows these changes.

Moreover, we notice that Avast and Norton sometimes do not quarantine the undetected root program (e.g.
W32.Virut.CF) which continues infecting other files that are detected and quarantined by the AVs. This causes
several applications to malfunction because they depend on the quarantined files.

100 Undetected malware (within

90 zero day) with no child files
80
70 Undetected malware (within
% of Malware

60 zero day) with undetected child

files
50
40 Undetected malware (within
30 zero day) with one or more
detected child files more than 0
20
day
10 Undetected malware (within
0 zero day) with one or more
detected child files within 0 day

Figure 4. The percentage of the malware in the malware collection that is not detected within zero days and also
produces one or more malicious child files

3.2.1.2 Registry
Registry contains a data structure that organizes configurations, called registry key. Changing values in registry
keys may affect system-wide or application-specific functionality and security. In the study, we observe which
registry keys malware attempts to modify and the AVs’ corresponding responses.

From the experiment, there is no record of denying registry modification in Avast’s and Symantec’s logs. However,
Symantec’s log contains the record of registry remediation. It implies that Symantec probably allows programs to
alter registry keys and remedy them later if a threat is found.

The results also suggest that the AVs with behavior-based detection provide protection for registry, although it
seems limited to system and network keys. These keys are often changed by malware. They control important
functions, such as automatically starting programs after reboot
(\Software\Microsoft\Windows\CurrentVersion\Run), controlling a browser's appearance and security, and setting
local/remote login (\Software\Microsoft\Windows NT\CurrentVersion\Winlogon). However, there are other keys
that malware can alter, leading to security breaches, such as Microsoft Word’s key controlling macro execution
(Software\Microsoft\Office\11.0\Word\Security\Level) and Microsoft Excel’s key controlling add-ins
(Software\Microsoft\Office\11.0\Excel\Resiliency\StartupItems). Additionally, there are other keys which do not
directly weaken system security but can help malware hide the fact of system being compromised. For example,
W32.IRCBot sets /SOFTWARE\Microsoft\Security

Digital Object Indentifier 10.1109/MC.2010.187 0018-9162/$26.00 ) 2010 IEEE

 This article has been accepted for publication in Computer but has not yet been fully edited.
Some content may change prior to final publication.

Center\[FirewallDisableNotify/AntiVirusDisableNotify/UpdatesDisableNotify] to disable a notification window

alerting users of firewall, antivirus or updates being disabled.

The results also show that the AVs do not always protect the important keys. For example, Norton and Trend
Micro allow 56.93% and 53.62% of the undetected malware which attempts to modify
\Software\Microsoft\Windows\CurrentVersion\Run, while Kaspersky, McAfee and Trend Micro deny 40.38%,
25.98% and 5.80% to perform the registry-key modification respectively. Similar results also occur for
\Software\Microsoft\Windows NT\CurrentVersion\Winlogon.

3.2.2 Memory
Malware commonly uses program-vulnerability exploitation and code injection to manipulate running programs to
perform malicious tasks (e.g. downloading other malware, obscuring malware’s presence). In the study, we
observe which exploitations the AVs detect, how long it takes for detection, and the AVs’ responses to malicious
code injection.

The empirical results show that additional exploitations are present during the undetected malware’s execution
and the AVs’ detection can lag behind even after vulnerabilities are published. For example, several undetected
malware files visit malicious web pages and download malicious images. McAfee, Norton and Symantec detect the
malicious images which attempt to create a vulnerable ActiveX object (HTTP MS MPEG2TuneRequestControl
ActiveX BO). McAfee and Trend Micro detect the web page that attempts to exploit Microsoft Data Access
Component vulnerability (Exploit-MS06-014). Some vulnerabilities may take the AVs days or weeks to detect after
publication. One example is that Norton detects HTML files that try to create a vulnerability-prone object (MSIE
ADODB.Stream Object File Installation Weakness) and exploit the buffer overflow vulnerability in a browser (HTTP
Microsoft IE Generic Heap Spray BO) about 30 days after we initially download the malware (mid August). These
two vulnerabilities were published in 2006 and July 2009 respectively. One way to mitigate the vulnerability-
exploitation problem is to keep both operating system (Windows) and applications up-to-date.

Additionally, the AVs do not always block malware from performing code injection. From the experiment,
Kaspersky and Trend Micro allow 21 and 19 malicious programs to perform code injection respectively. Norton and
Trend Micro block 51 and 74 malicious programs from performing code injection. There is no record of code
injection detection in Avast’s, McAfee’s and Symantec’s logs. This suggests that Avast, McAfee and Symantec do
not protect other programs against code injection.

3.2.3 Network
Network connection is one way for programs in the system to communicate with remote hosts as well as for
remote intruders to inject malware into the system. We study the mechanisms that the AVs use to secure network
communication and quantify the amount of malware the AVs prevent from downloading additional malicious files.

The AVs offer several features to secure network-related applications. McAfee, Norton and Trend Micro have
browser plug-ins to protect against browser exploitations and malicious web sites. All AVs have an email scanner to
filter malicious emails, although we find that Norton, Symantec and Trend Micro stop only a relatively small
amount of spam generated by malware.

The AVs with behavior-based detection also have their own firewall which controls both incoming and outgoing
traffic, while the default Windows firewall only monitors incoming traffic. However, these firewalls are not
sufficient to protect against certain threats because they do not examine actual traffic data. For example, Trojan
downloader can transfer malicious files using protocols permitted by the firewall (TCP/HTTP). However, these
threats cannot evade behavior-based detection. Our experiments show that Norton detects malware sending HTTP

Digital Object Indentifier 10.1109/MC.2010.187 0018-9162/$26.00 ) 2010 IEEE

 This article has been accepted for publication in Computer but has not yet been fully edited.
Some content may change prior to final publication.

requests to download malicious files, malware using an email with links to a copy of itself, and malware
downloading malicious PDF and CSS files with executable content. Moreover, Kaspersky, McAfee, Norton and
Trend Micro detect denial-of-service attacks, such as ICMP flood and TCP SYN flood.

Additionally, the AVs with behavior-based detection can stop undetected malware from downloading malicious
files. Kaspersky, Norton and Trend Micro prevent 30.51%, 74.38%, and 71.54% of the undetected malware with
malicious-web download from downloading one or more malicious files respectively. Malware is considered to
exhibit malicious-web download if at least one AV blocks one or more web sites visited by the malware. There is no
record of blocking malicious download in Avast’s, McAfee’s and Symantec’s logs. The amount of the undetected
malware with malicious-web download is 147 (13.18% of the malware). Trend Micro seems to block malicious
downloads based on blacklists, but Kaspersky and Norton block web sites that actually deliver suspicious files.
Hence, Trend Micro blocks web sites that are no longer available.

4 Conclusion
We study the effectiveness of six commercial AVs (with and without behavior-based detection) in defending
against contemporary malware. They quarantine at most 62.15% of the malware within the first day the malware
files are collected. 8.52%-21.88% of the malware are detected after 30 days. It is also likely that undetected
malware’s execution will result in additional malware in the system. On average 53.83% of the undetected
malware produce one or more malicious child files. 23.49% of the undetected malware with malicious child files
generate one or more child files detected immediately by the AVs, and an additional 64.36% produce child files
that are not detected by the AVs. During undetected malware’s execution, the AVs with behavior-based detection
can provide protection against certain system activities, as opposed to none by the AVs without behavior-based
detection. The results show that the AVs with behavior-based detection can prevent a subset of the malware from
modifying certain system and network registry values. They sometimes prevent malware from performing code
injections and malicious downloads (non-executable files (JPG/CSS) with executable content). 13.18% of the
malware is undetected and exhibit malicious-web downloads. Out of these malware files, they can stop 0%-74.38%
from downloading additional malware.

The empirical results suggest that regardless of having behavior-based detection, the AVs cannot effectively detect
current malware. Nonetheless, the AVs with behavior-based detection raise the bar for system protection. To
minimize the risk of receiving malware, users must take precautions before downloading or opening any unknown
files, such as not opening emails from strangers or opening unknown web sites, and always keeping the system
(both Windows and applications) up-to-date.

References
1. Christodorescu, M. and Jha, S. 2004. Testing malware detectors. SIGSOFT Softw. Eng. Notes 29, 4 (Jul. 2004), 34-44.
2. Stephanie Forrest , Steven A. Hofmeyr , Anil Somayaji , Thomas A. Longstaff, A Sense of Self for Unix Processes,
Proceedings of the 1996 IEEE Symposium on Security and Privacy, p.120, May 06-08, 1996.
3. Slowinska, A. and Bos, H. 2009. Pointless tainting?: evaluating the practicality of pointer tainting. In Proceedings of
the 4th ACM European Conference on Computer Systems (Nuremberg, Germany, April 01 - 03, 2009). EuroSys '09.
ACM, New York, NY, 61-74.
4. Martignoni, L., Stinson, E., Fredrikson, M., Jha, S., and Mitchell, J. C. 2008. A Layered Architecture for Detecting
Malicious Behaviors. In Proceedings of the 11th international Symposium on Recent Advances in intrusion Detection
(Cambridge, MA, USA, September 15 - 17, 2008). R. Lippmann, E. Kirda, and A. Trachtenberg, Eds. Lecture Notes In
Computer Science, vol. 5230. Springer-Verlag, Berlin, Heidelberg, 78-97

Digital Object Indentifier 10.1109/MC.2010.187 0018-9162/$26.00 ) 2010 IEEE

 This article has been accepted for publication in Computer but has not yet been fully edited.
Some content may change prior to final publication.

5. Christodorescu, M., Jha, S., and Kruegel, C. 2008. Mining specifications of malicious behavior. In Proceedings of the 1st
Conference on india Software Engineering Conference (Hyderabad, India, February 19 - 22, 2008). ISEC '08. ACM, New
York, NY, 5-14.
6. Gao, D., Reiter, M. K., and Song, D. 2004. Gray-box extraction of execution graphs for anomaly detection. In
Proceedings of the 11th ACM Conference on Computer and Communications Security (Washington DC, USA, October
25 - 29, 2004). CCS '04. ACM, New York, NY, 318-329.
7. Bhatkar, S., Chaturvedi, A., and Sekar, R. 2006. Dataflow Anomaly Detection. In Proceedings of the 2006 IEEE
Symposium on Security and Privacy (May 21 - 24, 2006). SP. IEEE Computer Society, Washington, DC, 48-62
8. Cha, S., Moraru, I., I., Jang, J., Truelove, J., Brumley, D., and Andersen, D. SplitScreen: Enabling Efficient, Distributed
Malware Detection. In Proc. 7th USENIX NSDI, (San Jose, CA), Apr. 2010.
9. Moser, A., Kruegel, C., and Kirda, E. 2007. Exploring Multiple Execution Paths for Malware Analysis. In Proceedings of
the 2007 IEEE Symposium on Security and Privacy (May 20 - 23, 2007). SP. IEEE Computer Society, Washington, DC,
231-245
10. C. Kruegel, E. Kirda, D. Mutz, W. Robertson, and G. Vigna, "Polymorphic worm detection using structural information
of executables", In Proceedings of Recent Advances in Intrusion Detection (RAID), 2005.
11. Yin, H., Song, D., Manuel, E., Kruegel, C., Kirda, E.: Panorama: Capturing system-wide information flow for malware
detection and analysis. In: Proceedings of the 14th ACM Conferences on Computer and Communication Security (CCS
2007) (2007)
12. Sangpetch, A., Turner, A., Kim, H. VManager: Service-aware virtual machine management. CMU-ECE-2009-09, 2009.

Digital Object Indentifier 10.1109/MC.2010.187 0018-9162/$26.00 ) 2010 IEEE