You are on page 1of 116

1

Live Forensics Tutorial


Part 2: Live Forensics
Frank Adelstein, Ph.D.
Technical Director, Computer Security, ATC-NY
GIAC-certified Digital Forensics Investigator

Golden G. Richard III, Ph.D.


Professor, Dept. of Computer Science, University of New Orleans
GIAC-certified Digital Forensics Investigator
Co-Founder, Digital Forensics Solutions, LLC
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Goals

Whats happening now


Who is doing what
Obtain another piece of the puzzle
Help focus static analysis (what parts of the disk)
Get passwords, unencrypted data, etc.
Information can be used to:
Reconstruct sessions (e.g., web, ftp, telnet, IM)
Find files (downloaded or accessed through network
drives)
Find passwords
Identify remote machines

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Why Live Forensics?


Big disks
Disk capacity keeps increasing (Oct06: 500Gb for ~$158) faster
than processors
Terabyte systems are big and common
Searching (or indexing) takes time
Mirroring takes time

Minimal downtime (mission critical sys)


Harder to seize systems (even with court order)
Provide context for static analysis
Low-profile examination
Long data lifetimes
Some data is only in RAM

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Why: The Lifetime of Volatile Data


Chow (Shredding your Garbage paper, see
references)
Lifetime of volatile data

usernames
passwords/encryption keys
credit card numbers
private conversations

Forensics analysis of physical memory reveals


private data, even weeks after use
Both a forensics and a privacy issue
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Lifetime (2)
Most current systems take no steps to overwrite
sensitive data in memory
Most applications that handle sensitive data werent
specifically designed to deal with sensitive data
e.g., Word processors:
Financial data, medical records, lists of passwords

Password entered into web browser may be duplicated


dozens of times
kernel buffers, window manager, application buffers, dynamic
memory allocator

If software crashes, core dump may leak this information


Live forensic analysis can reveal sensitive data months
after last use
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Chow: Data Lifetime


Ideal Lifetime: period of time data is actually in
use
first write after allocation last read before
deallocation

Natural Lifetime: period of time that data


remains readable
first write after allocation first write after
reallocation (first overwrite)

Secure Deallocation Lifetime: zero on


deallocation
first write after allocation deallocation (when it is
zeroed)
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Lifetime Experiment

Linux machine, 1GB RAM


Windows machine, 1GB RAM
Linux server, 256MB RAM
Linux version of experiment:
64MB buffer filled with 20 byte stamps
Allocate and release
Each stamp:
magic number, serial number, checksum

Windows version:
4MB buffer transmitted between machines
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Lifetime Experiment (2)


Immediately after allocation:
2-4MB of stamps remain

14 days later:
23KB 3MB of stamps remain

Additional 14 days on Linux server:


7KB of stamps remain

Most remaining stamps trapped in blocks of


memory in the kernel slab allocator
Reboot on Thinkpad T30 laptop: stamps
remain after 30 seconds w/o power
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Lifetime Experiment (3)

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

10

Persistence of (Post-Reboot) Memory


Research conducted at the
University of New Orleans
and ATC-NY
Many systems retain at least some data after a
warm reboot, reset, or even cold reboot
Highly dependent on model and BIOS settings
Potentially useful as a last resort for obtaining
live forensics data, assuming computer model is
known to have post-reboot persistent memory
Work still on-going
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

11

Live Forensics: Constraints


Minimize changes and artifacts
analogy to physical data
must balance tool footprint to usefulness

Timely evidence acquisition and analysis


Need access to the system
Need to minimize impact on system,
particularly if its mission-critical

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

12

Live Forensics: Dangers


Most live forensics tools rely on the OS to
provide some basic services
Reads of physical memory
Disk dumping

User-level rootkits
Shallow trickery
Modify system commands (e.g., ls, ps, du, df)
Change disk space statistics, list of running
processes, etc.

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

13

Live Forensics: Dangers (2)


Kernel-level rootkits
Deep trickery
Modify OS to produce arbitrary results
Allows files, blocks of physical memory, etc. to be
hidden even if trusted executables are used

Disk drivers
Hacking virtual memory system
Replacement of shared libraries
Affects even your trusted executables unless
theyre statically linked!

Specific examples a bit later


USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

14

Minimize Changes and Artifacts


Small footprint (using trusted software)
Try not to change anything
But everything changes, all the time
Minimize changes to evidence
Record all steps taken and artifacts

Low profile, minimize detection


Artifacts can be explained
analogy: detectives fingerprints on ransom note
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Timely Evidence Acquisition


and Analysis

15

First response/triage
Looking for evidence, or for what
computers, disks, directories, etc. may
contain evidence. Examples:
30 servers, search warrant says image 1
search warrant for quick hash scan to find
sufficient cause to get a broader warrant

Looking for context for static analysis


USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

16

Need Access to the System


Before investigation
Use preinstalled agents
Requires prior access (plan to be attacked?)
Agents must not have become compromised

During investigation
Need credentials for log-in
Must use known good binaries
See previous on artifacts
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

17

Typical Scenario

Triage/quick peek
Justify larger warrant
Ongoing crime (in progress)
Running an illegal server
Trojan horse defense (support/refute)
Whats going on with this machine?
Machine running slow
Lots of suspicious disk or network activity

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

18

Scenario: Triage
Limited time. Want to answer:
Is there a problem?
What machine(s) are affected?
What disks need to be imaged?
What is running on the system?

Focus on where the problem is the worst


and the evidence is the most abundant

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

19

Scenario: Justify Warrant


Non-disruptive, quick hash search
Look for the presence of a file or tools
Match against known hashes
Look for email addresses, etc.

Investigator does not take machine down,


disrupt service, or raise suspicion
Investigator does not see the actual files
If justified, can return for full investigation
Risk of evidence damage, must be careful
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

20

Scenario: Ongoing Crime


Want to catch them in the act
Show how things change (web pages, file
access times, registry, memory, etc.)
Want to understand:
How they got in
What they compromised
Where they are
Who they are
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

21

Scenario: Illegal Server


Compare whats running on the machine and
what ports are open to a network scan (e.g.,
nmap et al)
What services does the world see visible on the
computer?
Some techniques are subtle (port knocking),
but most are not
Look at network traces to see who is talking to
the computer (be careful of legal issues)
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

22

Scenario: Trojan Defense


Its not my fault, someone else is controlling my
computer!
Support or refute the claim

Are there traces of (known) Trojans?


Are there unusual network connections?
What is running on the machine (that can be seen)?
Any indication that something is hidden?
Enough other evidence/activities to corroborate?

Still new (1st used in court in UK in 2003)


Still tricky (often Trojans/viruses are present)
Plus, malware is often present that didnt impact
the case in a significant way
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

23

Information Available

Running processes
Open files
Network connections
Memory (physical / virtual dumps)
Regular disk files

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

24

Information Available (2)


Images of entire disk
Live disk imaging
(a.k.a. shooting a moving target)

Deleted files
Live file carving

Unencrypted document fragments


Encryption keys for whole-disk encryption
schemes
Copies of volatile-only malware (for disassembly
and investigation)
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

25

Running Processes
Windows

Open files
Open network connections
Registry activity
Open DLLs

Unix

Open files
Open network connections
Access to corresponding EXE, even if deleted
Command line that invoked application
Environment variables

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

26

Memory
Process memory
Finer-grained than dumping entire RAM
Easier to make sense of virtual address space for a
process than physical memory
More likely to find contiguous application structures
Can yield passwords, document fragments,
unencrypted documents

Kernel memory
Search for hidden processes
Evaluate health of kernel

String searches
Most brute force technique
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

27

Disk
Extract regular files
One difference from traditional:
Even for volatile distributions, e.g., Knoppix

Live disk imaging (moving target?)


Live file carving to retrieve deleted files
Need to be more creative than in traditional
case
Trickier to attach sufficient high-speed storage
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

28

Virtual Machines
Freeze VM
Copy/snapshot disks, memory, even screen
Resume execution

Can also dump memory, disk images over


local network connection between host
and VM while VM runs
nc -l -p 7000 -w 1 > dump.dd (host)
dd if=/dev/sda bs=512 | nc
192.168.1.50 7000 (within VM)
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

29

Analysis
What's going on?
Are things not right, processes or files
hidden, or disk encryption in use?
What's lingering in memory (a lot) and
process memory dumps?
Dumping OS structures
(e.g., determining which areas of swap space
correspond to a particular process)
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

50

Integrated Live Analysis Toolset


OnLine Digital Forensic Suite
Tools for live investigation, data acquisition,
and analysis
Web-based interface

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Live Forensics

USENIX Security 2007

51

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Create Inquiry

USENIX Security 2007

52

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Create (2)

USENIX Security 2007

53

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Target Information

USENIX Security 2007

54

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Confirm Information

USENIX Security 2007

55

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Target Password

USENIX Security 2007

56

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Initial Acquire

USENIX Security 2007

57

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Initial Acquire (2)

USENIX Security 2007

58

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Initial Acquire Complete

USENIX Security 2007

59

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

General System Information

USENIX Security 2007

60

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

IP Interface Information

USENIX Security 2007

61

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Data Analysis

USENIX Security 2007

62

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

SMB file server

Network Ports

63

vmware phoning
home to check
for updates

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Running Processes

USENIX Security 2007

64

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

65

Detailed Process Information

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

66

Network Connections by Process

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Another Process

USENIX Security 2007

67

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Open File Handles

USENIX Security 2007

68

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Software-based Acquisition of
Physical Memory

69

dd command
Under Windows, use \\.\PhysicalMemory device
(Not usable in user-space in XP SP2 +)
Under Linux:
/dev/mem

(problematic now)

Physical memory

/proc/kcore
Kernel virtual memory

Problem: must rely on OS to provide physical


memory dump
OS might be compromised
After acquisition, interpretation is another issue!
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

70

Software-Based (2)
Virtual machines (e.g., VMWare)
State of a virtual machine, including memory and
disk, can be extracted
Primary drawback is that the machine under
investigation must be a virtual machine

Hibernation files
Snapshot of physical memory
Can hibernation process (e.g., writing of physical
memory) be subverted?
Probably. Unaware of available tools to do it now.
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

General Problems with Memory


Acquisition

71

Software-based solutions for memory


imaging typically require loading software
Probably erases some evidence
Requires at least limited trust of the OS
Hardware solutions generally require preinstalled cards
Potentially subject to nasty hardware
poisoning tricks
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Categories for Anti-Acquisition


Attacks

72

DoS Attack
Crash/Halt machine when somebody tries to acquire RAM
Can have legal consequences for investigator

Covering Attack
Acquisition tool can not read some part of physical memory
instead it reads garbage (e.g. 0x00 bytes).
CPU sees the real content, which e.g. may contain malicious
code and data

Full Replacement Attack


Like Covering Attack, but the attacker can also provide custom
contents (instead of garbage) for the acquisition tool

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III


Used with permission
of Joanna Rutkowska

73

DoS Attack

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III


Used with permission
of Joanna Rutkowska

74

Covering Attack

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III


Used with permission
of Joanna Rutkowska

75

Full Replacement Attack

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III


Used with permission
of Joanna Rutkowska

76

Shadow Walker

Described in a paper by Sparks (see references)


Details somewhat complicated
Look away if necessary
Relies on split translation lookaside buffers
(TLB) on IA-32 Intel processors
TLB for data accesses
TLB for instruction accesses
Shadow Walker poisons TLB using a hacked
page fault handler
Desynchronizes code and data TLBs
Means: can hide code of executing processes

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Aside: Virtual Memory


Implementation (Abstract)

77

Split virtual address into page number (p)


and page offset (d)
virtual address

CPU

physical memory

f
Single-level page
table shown for
simplicityon modern
architectures, would
generally be multi-level
USENIX Security 2007

.
.
page table

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Speeding up Virtual Memory: TLB


(Translation Look-aside Buffer)

78

TLB is a block of associative memory for


caching previous page table lookups
On-chip and extremely fast
Search for physical memory frame numbers in
parallel
Page #

virtual
address

USENIX Security 2007

Frame #

TLB

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

79

Split TLBs on IA-32


Page #
virtual
address
for
instruction
fetch

Code
TLB

Frame #

12

99

15

72

Page #
virtual
address
for read /
write

USENIX Security 2007

Data
TLB

!!

Frame #

12

17

18

82

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

80

TLB De-synchronization
page # 12
000100100100
000000010010
010011111010

Malware executing in virtual


page # 12
frame # 17

frame # 99
Code reading page # 12 to check for
malware sees a different virtual physical mapping
USENIX Security 2007

4 oz. butter
2 cups sherry vinegar
2 cups vintage port
tsp Chinese 5 spice

000100100100
000000010010
010011111010
physical memory

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

81

Shadow Walker (2)


Hidden processes execute w/ no problem
Read accesses to targeted virtual addresses
are trapped and diverted
No read access to code for targeted executing
processes
Again, not hopelessmay be able to, e.g.,
validate page fault handler
Other restrictions, e.g., hidden pages must be
non-paged yet are marked non-resident, etc.
Full replacement attack against softwarebased acquisition
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

82

Hardware-based Memory Dumping


Hardware-based Memory Acquisition:
Firewire
Earlier: Maximillian Dornseif (see references)
More robust: Adam Boileau (metlstorm),
raw1394 stuff
Security-Assessment / Immunity

Tribble
Carrier (see references)

Related:
Copilot: system monitoring
GPU-based system protection
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

83

Tribble Hardware Design

USENIX Security 2007

Source: Carrier paper

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

84

PCI Bus

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Source: Carrier paper

85

Acquisition Process (Simplified)


PCI card is powered on and initializes
PCI controller is disabled and card
remains idle until acquisition is initiated
When the switch is activated:
External storage device is activated
PCI controller is enabled
Target CPU is halted
Log is created on storage device, including
date/time of acquisition
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

86

Acquisition Process (Simplified) (2)


When the switch is activated (cont.):
Loop, beginning at physical memory address 0:
Read X bytes of memory using DMA. If areas are inaccessible,
substitute 0s and make appropriate entries in the log
If an I/O error occurs, then create an appropriate entry in the log and
substitute 0s
If no I/O error occurs, then acquisition card writes the buffer in
SDRAM to the memory image on the external storage device
Consider X bytes read in calculation of cryptographic hash for
memory image

Write cryptographic hash value in log on external storage


Write log entry indicating that acquisition has ended and include
date/time
Disable PCI controller
Disable external storage device
Acquisition card returns to idle state
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

87

Status of Tribble
Minimal prototype of PCI-based memory
acquisition device
Proof of concept doesnt include external
storage, button, etc.
But idea is sound
Major limitation: PCI card must be installed
before acquisition
Most appropriate for corporate environments
Firewire hacks dont require any pre-installed
hardware, but are more fragile
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

88

Subverting HW Memory Acquisition


HW-based memory acquisition is also
subject to attack, just like SW approaches
Interesting attack described by Joanna
Rutkowska (invisiblethings.org) at
Blackhat DC 2007
Currently targets only AMD-based
systems

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

89

AMD System

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III


Used with permission
of Joanna Rutkowska

90

HW Access to Physical Memory

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III


Used with permission
of Joanna Rutkowska

91

DoS Attack
Hack AMD Northbridge memory
configuration registers
Map memory addresses accessed
by PCI devices back to I/O
address space
Access to targeted addresses
hangs system

USENIX Security 2007

Based on a
slide2007
bybyJoanna
Copyright
Frank AdelsteinRutkowska,
and Golden G. Richard.used
III
with permission

92

Covering Attack
Enhance DoS attack by
programming unused PCI bridge
to respond to range of memory
addresses
Now system doesnt hang, but
reads by PCI devices will return
cover values (0xFF in JRs
experiments)

USENIX Security 2007

Based on a
slide2007
bybyJoanna
Copyright
Frank AdelsteinRutkowska,
and Golden G. Richard.used
III
with permission

93

Full Replacement Attack?


Full replacement attack against HW-based
acquisition is much harder
Basic idea only:
Instead of mapping addresses to unused PCI
bridge
Map addresses to device memory on some PCI
device
e.g.: device memory on graphics card
Then, can provide arbitrary cover values!

This sort of attack will be extremely fragile


Good news for investigator
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

94

Assumption Weakening Malware


Weve examined a few types of attacks that weaken an
investigators ability to capture system state
This kind of malware also complicates establishment of
causal relationships
Not just accused didnt download thata virus did it
Enables stealthy insider attacks
Prevents investigator from eliminating suspects based
on limited access rights
But these kinds of attacks arent very common
Tend to be fragile
Awareness of the possibility of the attacks is the
most important thing for a typical investigation
One more
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

95

System Management Mode Attack


Quick Aside: Intel x86 CPU modes:
Real mode
16-bit mode used during boot or for running legacy OSs

Protected mode
Typical mode used in modern operating systems
Supports hardware memory protection, paging, etc.

Virtual 8086 mode


Used primarily for running legacy DOS applications

System Management Mode


Special mode for performing chipset-level management
functions (OS independent)
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

96

System Management Mode


System management mode (SMM) entered upon
SMI interrupt
Can only be entered via this interrupt
Generated by chipset, intended for use by
applications in firmware
General uses:
Execution of code for temperature critical situations
Executed upon lid closure in a laptop for power
savings
Unused disk spin down, other out of band execution

OSs and application software are typically


completely unaware of SMM switches
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

97

System Management Mode


Processor state is completely saved on
entry into SMM
SMM code resides in a separate address
space (SMRAM)
In SMM, everything is wide open
No memory or hardware access restrictions
All interrupts disabled
Can only exit SMM via the RSM instruction

SMM detailed extensively in Intel 64 and


IA-32 Architectures Software Developers
Manual (Ch 24)
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

98

SMM Attack: Basic Idea


Paper by Duflot (see references)
On some OSs, such as OpenBSD, administrator
account may be limited
Value of securelevel dictates what privileges
account has
> 0 active, higher number == lower privilege
e.g., at higher levels, cannot override IP packet filtering rules

-1 disabled (insecure mode)

Use SMM to hack securelevel mode and add


privileges to root account
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

99

SMM Attack: Hurdles


Need to change code executed on SMI
SMRAM typically overlaps video RAM and so memory accesses
are sent to video card unless already in SMM
Solution: set D_OPEN bit of SMRAM control register in the
North Bridge
Directs memory accesses to SMRAM
Now write physical memory to change SMM handler
D_LCK bit prevents any other bits in the SMRAM control register
from being changedif this was already set in the boot
sequence, no luck

Need to cause SMI to be raised


Access control register of Advanced Power Management

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

100

SMM Attack: More Background

X server under *BSD needs to access physical memory


Access allowed through /dev/xf86 if allowaperture <> 0
Only one process allowed to access /dev/xf86
Kill X server
Hack SMM registers to allow modification of SMRAM
Then install new handler:
changes value of the securelevel variable using physical
memory address
This obtainable through:
nm /bsd | grep securelevel - 0xd0000000

User now has unlimited privileges as root


USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

101

SMM Attack: Impact


Can invalidate assumptions about what
legitimate users of a machine were able to do:
User didnt have physical access to machine
So couldnt have rebooted the machine using a CD to gain
complete control

User was an administrator, but securelevel setting


prevented direct access to disk devices
User couldnt have changed on-disk structures directly
to incriminate his co-worker

Oops!
Punchline: Its not that the attack cant be
detected
But detection takes time and resources
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

102

SMM Attack: Prevention


Some basic ideas:
Dont run X on critical systems
Allows allowaperture = 0, which disallows access to
physical memory
*BSD install process is smartif you say you wont
run X, allowaperture is set to 0 and this attack is
impossible

Have the boot process set the D_LCK bit


Then cant manipulate SMM registers to set up attack

Most important: Be aware of the potential for


attacks like this
They arent common, but as an investigator, you
dont want to be surprised
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

103

Analyzing Memory Dumps

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

104

Memory Dump Analysis


Assuming that a trusted dump of system memory has
been obtained, now what?
Analyze dump to extract information about processes,
threads, open files, sockets, etc.
Most interesting things in the kernel are objects (e.g.,
structures)
These objects likely have many pointers hanging off
First method: analyze lists/tables of kernel structures
From symbol table for kernel, determine location of interesting
tables/lists and enumerate
e.g., for Linux, System.map file created when kernel is compiled
Can locate system call table, first process, etc.

Second method: do carving for interesting objects


USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

105

FATKIT / Volatools
Python-based system for examining physical memory dumps
C:\VolatoolsBasic-1.1.1>python volatools
usage: volatools cmd [cmd_opts]
Supported Commands:
connections
datetime
dlllist
files
ident
modules
pslist
sockets
strings
vaddump
vadinfo
vadwalk
USENIX Security 2007

Print list of open connections


Get date/time information for image
Print list of loaded dlls for each process
Print list of open files for each process
Identify image properties such as DTB and VM type
Print list of loaded modules
Print list of running processes
Print list of open sockets
Match physical offsets to virtual addresses
Dump the VAD sections to files
Dump the VAD info
Walk the VAD tree
Copyright 2007 by Frank Adelstein and Golden G. Richard. III

106

C:\VolatoolsBasic-1.1.1>python volatools ident -f d:\MEMDUMP.1GB

Image Name: d:\MEMDUMP.1GB


Image Type: XP SP2
VM Type: nopae
DTB: 0x39000
Datetime: Thu Mar 22 18:07:31 2007

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

C:\VolatoolsBasic-1.1.1>python volatools pslist -f d:\MEMDUMP.1GB


Name
System
smss.exe
csrss.exe
winlogon.exe
services.exe
lsass.exe
svchost.exe
svchost.exe
svchost.exe
svchost.exe
svchost.exe
spoolsv.exe
MDM.EXE
ntrtscan.exe
tmlisten.exe
OfcPfwSvc.exe
alg.exe
XV69C2.EXE
AcroRd32.exe
explorer.exe
jusched.exe
PccNTMon.exe
ctfmon.exe
reader_sl.exe
cmd.exe
USENIX Security 2007
dumpmem.exe

Pid
4
436
492
516
560
572
752
812
876
924
976
1176
1372
1416
1548
1636
2028
336
2452
840
2608
2184
3084
1240
368
2132

PPid
0
4
436
436
516
516
560
560
560
560
560
560
560
560
560
560
560
1416
848
3844
840
840
840
840
840
368

Thds
65
3
20
22
17
19
21
9
72
6
7
14
4
13
14
9
6
1
0
16
2
4
1
2
1
1

Hnds
262
21
421
626
366
405
214
264
1582
95
137
159
85
65
179
145
103
84
-1
410
36
67
70
35
30
17

107

Time
Thu Jan 01 00:00:00 1970
Thu Mar 15 08:04:12 2007
Thu Mar 15 08:04:13 2007
Thu Mar 15 08:04:14 2007
Thu Mar 15 08:04:14 2007
Thu Mar 15 08:04:15 2007
Thu Mar 15 08:04:15 2007
Thu Mar 15 08:04:16 2007
Thu Mar 15 08:04:16 2007
Thu Mar 15 08:04:16 2007
Thu Mar 15 08:04:16 2007
Thu Mar 15 08:04:17 2007
Thu Mar 15 08:04:25 2007
Thu Mar 15 08:04:25 2007
Thu Mar 15 08:04:28 2007
Thu Mar 15 08:04:29 2007
Thu Mar 15 08:04:32 2007
Thu Mar 15 08:04:34 2007
Wed Mar 21 03:53:27 2007
Thu Mar 22 23:05:51 2007
Thu Mar 22 23:05:54 2007
Thu Mar 22 23:05:54 2007
Thu Mar 22 23:05:54 2007
Thu Mar 22 23:05:55 2007
Thu
Mar 22 23:07:01 2007
Copyright 2007 by Frank Adelstein and Golden G. Richard. III
Thu Mar 22 23:07:30 2007

C:\VolatoolsBasic-1.1.1>python volatools sockets -f d:\memdump.bluelu 108


Pid
1828
4
736
468
196
1936
4
1828
1112
1804
384
384
4
1936
316
1164
468
1828
4
196
1936
4
1112

Port
500
445
135
1900
1031
1025
139
0
123
1029
1028
1032
137
1026
1030
3793
1900
4500
138
1037
1027
445
123

USENIX Security 2007

Proto
17
6
6
17
6
6
6
255
17
17
6
6
17
6
6
6
17
17
17
6
6
17
17

Create Time
Wed Mar 28 02:22:36
Wed Mar 28 02:22:20
Wed Mar 28 02:22:25
Wed Mar 28 02:22:58
Wed Mar 28 02:22:54
Wed Mar 28 02:22:35
Wed Mar 28 02:22:20
Wed Mar 28 02:22:36
Wed Mar 28 02:22:39
Wed Mar 28 02:22:37
Wed Mar 28 02:22:36
Wed Mar 28 02:22:56
Wed Mar 28 02:22:20
Wed Mar 28 02:22:35
Wed Mar 28 02:22:44
Wed Mar 28 02:22:28
Wed Mar 28 02:22:58
Wed Mar 28 02:22:36
Wed Mar 28 02:22:20
Wed Mar 28 02:23:03
Wed Mar 28 02:22:35
Wed Mar 28 02:22:20
Wed Mar 28 02:22:39

2007
2007
2007
2007
2007
2007
2007
2007
2007
2007
2007
2007
2007
2007
2007
2007
2007
2007
2007
2007
2007
2007
2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

C:\VolatoolsBasic-1.1.1>python volatools files -f d:\MEMDUMP.1GB


109
************************************************************************
Pid: 4
File
\Documents and Settings\Administrator.HE00\NTUSER.DAT
File
\Documents and Settings\Administrator.HE00\NTUSER.DAT.LOG
File
\System Volume Information\_restore{1625C426-0868-4E67-8C2125BB305F7E1E}\RP228\change.log
File
\Topology
File
\pagefile.sys
File
\WINDOWS\system32\config\SECURITY
File
\WINDOWS\system32\config\SECURITY.LOG
File
\WINDOWS\system32\config\software
File
\WINDOWS\system32\config\software.LOG
File
\hiberfil.sys
File
\WINDOWS\system32\config\system
File
\WINDOWS\system32\config\system.LOG
File
\WINDOWS\system32\config\default
File
\WINDOWS\system32\config\default.LOG
File
\WINDOWS\system32\config\SAM
File
\WINDOWS\system32\config\SAM.LOG
File
\Documents and Settings\NetworkService.NT AUTHORITY\NTUSER.DAT
File
\Documents and Settings\NetworkService.NT AUTHORITY\ntuser.dat.LOG
File
\
File
\Documents and Settings\LocalService.NT AUTHORITY\ntuser.dat.LOG
File
\Documents and Settings\LocalService.NT AUTHORITY\NTUSER.DAT
File
\WINDOWS\CSC\00000001
************************************************************************
Pid: 436
File
\WINDOWS
File
\WINDOWS\system32
Copyright 2007 by Frank Adelstein and Golden G. Richard. III

USENIX Security 2007

110

Following Lists Approach: More


Windows Memory Forensics Toolkit (wmft)
http://strony.aster.pl/forensics/

kntlist
MemParser
One challenge: DKOM

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

111

Direct Kernel Object Manipulation


Idea: kernel components have access to
kernel memory
(at least in non-microkernel OSs!)

Malicious kernel component can modify


kernel structures to hide, e.g., processes
Good discussion of DKOM here:
http://www.blackhat.com/presentations/winusa-04/bh-win-04-butler.pdf
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

112

FU Rootkit and Descendents


pid 17
Doubly-linked process list in Windows kernel

C:\> fu ph 17

Processes continue
to run because Windows
scheduler handles
threads, not processes

pid 17
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

113

Lists and DKOM


Not hopeless
For process hiding case, can look deeper than
the process list
Look at lists of threads, make sure they match up
with processes in the process list
More difficult, because offsets of important kernel
structures for this effort are version-specific?
Walk all process lists, including those used by
the scheduler
Walk handle lists
Supplement with carving approach (next)
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

114

Schuster Approach: Carving


Want to find all processes/threads in
memory dump
Normal
Hidden
Terminated

Dont rely on kernel lists/tables


Search memory dump for objects that look
like processes/threads
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

115

Schuster (2)
Important ideas:
Memory is needed to store kernel objects
Use info about how kernel performs allocation to find blocks
of allocated memory

Kernel objects have an OBJECT_HEADER structure


Further, processes and threads have a
DISPATCH_HEADER, used for
scheduling/synchronization

Use these ideas to develop templates for


discovering interesting structures in a Windows
memory dump
Walk memory dump in 4K steps
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

116

Schuster: POOL_TAG

PoolTag == 0xe36f7250 for processes


PoolTag == 0xe5726854 for threads

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

117

Schuster: OBJECT_HEADER
Known values
for live/dead
processes and threads!

Also know information about lengths associated with name.


USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

118

Schuster: Additional Tests


Know some characteristics of
DISPATCH_HEADER
Know some characteristics of ETHREAD
structures (e.g., pointers to owning
process, DISPATCH_HEADER w/ type
thread, )

For a certain Windows version, size field is constant for a particular object type.
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

119

Schuster: Results
Perl-based PTFinder
Visualization using Graphviz
More on thiswill play a role in the demo
later
For now, some screenshots

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

120

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

121

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

122

RAM Carving
Process dump of MSN Messenger yields chat
message fragments:
Content-Type: text/plain; charset=UTF-8
X-MMS-IM-Format: FN=MS%20Shell%20Dlg; EF=;
CO=0; CS=0; PF=0
Are you enjoying Mardi Gras this year? I hear that the
crowds are smaller, but that the general spirit is
high

Can construct patterns based on these


fragments and apply file carving techniques to
discover fragments of chat sessions days or
weeks old in memory dumps
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

123

Live Disk Imaging


Can image disks live using essentially the
same tools as for preservation step in
dead analysis
Problem: Moving target, files changing
On a relatively quiet system, image may
be a reasonable representation
Win: dd if=\\.\PhysicalDrive0 of=e:\pd0.dd
Linux: dd if=/dev/hdc of=/mnt/images/hdc.dd
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

124

Live File Carving


Similarly, can perform file carving against
live block devices using standard tools
e.g., Scalpel, Foremost
Beyond consistency problem, need
sufficient available storage
Next Generation In-place, live carving

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

125

In-Place File Carving


client applications
scalpel_fs

preview database

FUSE
nbd client

local drive

network
nbd server

Scalpel

remote drive

G. G. Richard III, V. Roussev, V. Marziale, In-Place File Carving, Research


Advances in Digital Forensics
Copyright 2007 by Frank Adelstein and Golden G. Richard. III
USENIX Security 2007
III, Springer, 2007

126

(Very) Hard Problems


Can you trust the O/S?
kernel level rootkits
can you get around itor under it?
can you know when you can trust?

Whole disk encryption


BitLocker, EFS, CFS, TCFS, sfs, etc.
pull the plug and then, ooops

What do you do with a memory dump?


beyond string searches
reconstruct processes (running and dead?)
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

127

The Future (maybe)


Smarter techniques for
traditional forensics
Single CPU traditional forensics impractical
due to huge disks
Need parallel traditional forensics supplemented
with live forensics
Live forensics will provide increasingly essential
information for investigations
Live forensics will be broadly accepted (in court)
Not just extraction of strings
Much better techniques for live forensics
USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

128

References
Papers and other
resources

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Traditional Forensics: Selected


Papers

129

K. Eckstein, M. Jahnke, Data Hiding in Journaling File Systems, Proceedings of the 5th Annual
Digital Forensic Research Workshop (DFRWS 2005), New Orleans, 2005.
S. Garfinkel, Disk Imaging with the Advanced Forensics Format, Library and Tools,"
Proceedings of the Second Annual IFIP WG 11.9 International Conference on Digital Forensics,
Orlando, FL, Jan 2006.
M. Geiger, Evaluating Commercial Counter-Forensic Tools, Proceedings of 5th Annual Digital
Forensic Research Workshop (DFRWS 2005), New Orleans, 2005.
L. Marziale, G. G. Richard III, V. Roussev, "Massive Threading: Using GPUs to Increase the
Performance of Digital Forensics Tools," Proceedings of the 7th Annual Digital Forensics
Research Workshop (DFRWS 2007), Boston, MA, 2007.
G. G. Richard III, V. Roussev, "Toward Secure, Audited Processing of Digital Evidence:
Filesystem Support for Digital Evidence Bags," Research Advances in Digital Forensics II,
Springer, 2006.
G. G. Richard III, V. Roussev, "Next Generation Digital Forensics," Communications of the ACM,
February 2006.
G. G. Richard III, V. Roussev, "Scalpel: A Frugal, High Performance File Carver," Proceedings of
the 2005 Digital Forensics Research Workshop (DFRWS 2005), New Orleans, LA.
V. Roussev, G. G. Richard III, "Breaking the Performance Wall: The Case for Distributed Digital
Forensics," Proceedings of the 2004 Digital Forensics Research Workshop (DFRWS 2004),
Baltimore, MD.
P. Turner, Applying a Forensic Approach to Incident Response, Network Investigation and
System Administration using Digital Evidence Bags," Digital Investigation, 4(2007), pp. 30-35.

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Traditional Forensics: Selected


Books

130

E. Casey, Digital Evidence and Computer Crime,


Academic Press, 2004.
M. Caloyannides, Privacy Protection and Computer
Forensics, Second Edition, Artech House, 2004.
B. Carrier, File System Forensic Analysis, AddisonWesley, 2005.
D. Farmer, W. Venema, Forensic Discovery, AddisonWesley, 2004.
H. Carvey, Windows Forensics and Incident Recovery,
Pearson Publications, 2004.

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

Traditional Forensics: Selected


Web Sites

131

http://www.dfrws.org
Lots of references related to digital forensics, including a link to an interesting ejournal

http://www.ijde.org/
International Journal of Digital Evidence

http://www.tucofs.com/tucofs/tucofs.asp?mode=mainmenu
Collection of forensics-related software

http://www.sleuthkit.org
Home of Sleuthkit and Autopsy tools

http://www.digitalforensicssolutions.com
Home of Scalpel (file carving software)

http://www.linux-ntfs.org/
The Linux NTFS Project

http://www.nongnu.org/gfzip
The Generic Forensic Zip Project

http://pyFlag.sourceforge.net/
PyFLAG: Forensics and Log Analysis GUI

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

132

Traditional: Selected Software


Commercial and open-source digital forensics software

Sleuthkit / Autopsy
Scalpel
Foremost
Encase
FTK (Forensics Tool Kit)
ILook (law enforcement only)
WinHex
lots more

Open source digital forensics software project


http://www.opensourceforensics.org/

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

133

Traditional with Network Focus

The Tao of Network Security Monitoring: Beyond Intrusion


Detection, Richard Bejtlich, Addison-Wesley, 2004
Intrusion Signatures and Analysis, Mark Cooper et al, SAMS, 2001
Network Intrusion Detection, Stephen Northcutt and Judy Novak,
2002
Books by W. Richard Stevens:
TCP/IP Illustrated, Volume 3: TCP for Transactions, HTTP, NNTP, and
the UNIX Domain Protocols, Addison-Wesley, 1996
TCP/IP Illustrated, Volume 2: The Implementation, Addison-Wesley,
1995
TCP/IP Illustrated, Volume 1: The Protocols, Addison-Wesley, 1994

tcpdump, www.tcpdump.org
Wireshark (aka Ethereal), www.wireshark.org
WinHex, www.winhex.com

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

134

Live Forensics: Selected Papers

F. Adelstein, Live forensics: Diagnosing Your System Without Killing It First, Communications of the ACM 49, 2
(Feb. 2006), pp. 63-66.
B. Carrier, Risks of Live Digital Forensic Analysis, Communications of the ACM 49, 2 (Feb. 2006), pp. 56-61.
B. Carrier, J. Grand, A Hardware-based Memory Acquisition Procedure for Digital Investigations, Digital
Investigation (2004):1.
E. Casey, Practical Approaches to Recovering Encrypted Digital Evidence, International Journal of Digital
Evidence, (2002) 1:3.
J. Chow, B. Pfaff, T. Garfinkel, K. Christopher, and M. Rosenblum, "Understanding Data Lifetime via Whole
System Simulation," Proc. of the 13th USENIX Security Symposium, August 2004.
J. Chow, B. Pfaff, T. Garfinkel, and M. Rosenblum, Shredding Your Garbage: Reducing Data Lifetime Through
Secure Deallocation, Proceedings of the 14th USENIX Security Symposium, 2005.
M. Dornseif, FireWire - all your memory are belong to us, http://md.hudora.de/presentations/.
Loc Duflot, Daniel Etiemble, Olivier Grumelard, Using CPU System Management Mode to Circumvent Operating
System Security Functions, Proceedings of CanSecWest, 2006.
S. Garfinkel, Forensic Feature Extraction and Cross-Drive Analysis, Proceedings of the 6th Annual Digital
Forensic Research Workshop (DFRWS 2005), West Lafayette, IN, 2006
N. Petroni, A. Walters, T. Fraser, and W. Arbaugh, "FATKit: A Framework for the Extraction and Analysis of Digital
Forensic Data from Volatile System Memory", Digital Investigation, 3(4):197-210, December 2006.
N. Petroni, T. Fraser, J. Molina, and W. Arbaugh, "Copilot - a Coprocessor-based Kernel Runtime Integrity
Monitor," Proc. of the 13th USENIX Security Symposium, August 2004.
A. Schuster, Searching for Processes and Threads in Microsoft Windows Memory Dumps, Proceedings of the
6th Annual Digital Forensic Research Workshop (DFRWS 2006), West Lafayette, IN, 2006.
G. G. Richard III, V. Roussev, L. Marziale, "In-place File Carving," Research Advances in Digital Forensics III,
Springer, 2007.
J. Rutkowska, Beyond The CPU: Defeating Hardware Based RAM Acquisition Tools (Part I: AMD case), Black
Hat DC 2007.
S. Sparks, J. Butler, Raising the Bar for Windows Rootkit Detection, Phrack Issue # 63.

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

135

Live Forensics: Selected Web Sites

www.invisiblethings.org
http://www.vidstrom.net/
http://www.usenix.org/events/sec05/tech/full_papers/chow/chow.pdf (14th
Usenix Security)
http://www.security-assessment.com/Presentations/Auscert_2006__Defeating_Live_Windows_Forensics_DB_v1.8.ppt
http://md.hudora.de/presentations/firewire/2005-firewire-cansecwest.pdf
http://forensic.seccure.net/
http://www.knoppix.net
http://www.gcn.com/print/25_22/41502-1.html (Special Report, Live
forensics is the future for law enforcement)
http://news.com.com/2100-7349_3-5092781.html (U.K. teen acquitted with
Trojan defense, Oct. 17, 2003)
http://www.newsmax.com/archives/articles/2003/8/12/204345.shtml (The
Trojan Horse Defense in Child Pornography, Aug. 13, 2003)

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III

136

END OF Live Analysis


NEXT: Putting it all together
(demo)

USENIX Security 2007

Copyright 2007 by Frank Adelstein and Golden G. Richard. III