Procedures in Intrusion Investigation of a UNIX/Linux Host

CS-585F: Co !uter-"e#ated La$ and Co !uter Forensics Fa## %&&%

'( )u-Li C*en "a+es* ,enon -oe ,es#ovic*

.a/#e of Contents Abstract Acknowledgements Introduction Signs of a Compromised UNIX/LINUX S stem !a s "o #reserve $vidence Investigation of a Linu& S stem Collection (f $vidence Anal sis (f $vidence Summar +ibliograp, iii iv 1 1 % ' ' 1) 1* 1'

ii

0/stract
UNIX and Linu& s stems are vulnerable to various forms of network attacks.epending on t,e diligence of t,e people administering t,ese s stems t,e can eit,er be e&tremel eas or e&tremel difficult to gain unaut,ori/ed access to- ",is document is a case stud on ,ow to perform an intrusion detection anal sis on a UNIX ,ost in general0 and more specificall a Linu& s stem-

iii

0c1no$#edge ents
!e would like to e&press our sincere t,anks to .r- Ab/ug for ,is ,elp on t,is pro1ect- !e would also like to e&tend t,anks to t,e +ridgewater College Information "ec,nolog Center for t,e use of t,eir facilities and e2uipment t,roug,out t,is process-

iv

Introduction
UNIX does not ,ave good reputation for reliabilit or securit 34allmann0 15556Alt,oug, UNIX does offer some effective securit features suc, as login and user accounts w,ic, are saved in t,e /etc/passwd file0 access control wit, a granularit of owner0 group0 and world0 and keep log files7usr/adm/lastlog, /var/adm/utmp, /var/adm/wtmp/, /var/adm/acctUNIX s stems directl connected to t,e Internet are often sub1ect to ,acking attempts 38ruse 9 :eiser0 );;)6",e skill and knowledge to investigate a compromised Uni& s stem and t,e abilit to respond to a computer securit incident ,as become essential for bot, UNIX users and forensic investigators",e contents of t,is document will begin wit, looking for t,e signs of a compromised UNIX s stem0 and t,en proceed to preserve and gat,er evidence- After collecting information and sei/ing t,e data we need0 t,e ne&t step is to anal /e t,e file s stem to look for modifications to data and review log files to e&amine signs of intrusion",is document will also include an intrusion investigation of a server owned b t,e A+C Corporation- ",e administrators of t,e s stem ,ave seen log evidence to suggest repeated attempts to gain access to t,e server from unaut,ori/ed ,osts- Included will be t,e steps t,at t,e investigators c,ose to take in t,is particular investigation0 and an anal sis of t,e outcome of t,eir efforts- A recurring concern of t,e investigators will be t,e fact t,at t,e server was a production server0 and t,at t,e administrators would not allow t,e server to be taken offline during t,e investigation-

Signs of a co !ro ised UNIX/Linux s(ste

1. Examine log files for connections from unusual locations, or other unusual activity. <efer to /etc/syslog.conf to see ,ow logging is configured for eac, s stem service and w,ere it is sent- +elow is a list of some of t,e more common UNIX and Linu& log file names0 t,eir function0 and w,at to look for in t,ose files messages ",e messages log in t,e /var/log director will contain a wide variet of information",e t,ings to look for in t,is file are anomalies- Also0 look for events t,at occurred around t,e known time of t,e intrusion xferlog If t,e compromised s stem ,as a functioning ftp server0 xferlog will contain log files for all of t,e ftp transfers- ",is ma ,elp ou discover w,at intruder tools ,ave been uploaded to our s stem0 as well as w,at information ,as been downloaded from our s stem utmp

",is file in t,e /var/log director contains binar information for ever user currentl logged in- ",is is onl useful to determine w,o is currentl logged in- A wa to access t,is data is eit,er t,e who command0 or t,e w commandwtmp $ver time a user successfull logs in0 logs out0 or our mac,ine reboots0 t,e wtmp file in t,e /var/log director is modified- ",is is a binar file= t,e tool used to obtain useful information is lastsecure Some versions of UNIX 3<ed:at Linu& for e&ample6 log tcp wrapper messages to t,e secure log file in t,e /var/log director - $ver time a connection is establis,ed wit, one of t,e services running out of inetd t,at uses tcp wrappers0 a log message is appended to t,is log file- !,en looking t,roug, t,is log file0 look for anomalies suc, as services t,at were accessed w,ic, are not commonl used0 or connections from unfamiliar ,osts-

2. Look for setuid and setgid files (especially setuid root files everywhere on your system. Intruders often leave setuid copies of /!in/sh or /!in/time around to allow t,em root access at a later time- ",e UNIX find program can be used to ,unt for setuid and/or setgid files- >or e&ample0 ou can use t,e following commands to find setuid root files and setgid kmem files on t,e entire file s stem?
# find / -user root perm 4000 print # find / -group kmem perm 2000 print

". #heck your system !inaries to make sure that they have not !een altered. Intruders ma c,ange e&ecutable programs on UNIX s stems suc, as login, su, telnet, netstat, ifconfig, ls, find du, df, li!c, sync, an binaries referenced in /etc/inetd.conf, and ot,er critical network and s stem programs and s,ared ob1ect libraries- Compare t,e versions on our s stems wit, known good copies0 suc, as t,ose from our initial installation media"ro1an ,orse programs ma produce t,e same standard c,ecksum and timestamp as t,e legitimate version- +ecause of t,is0 t,e standard UNIX sum command and t,e timestamps associated wit, t,e programs are not sufficient to determine w,et,er t,e programs ,ave been replaced- ",e use of cmp, @.A0 "ripwire0 and ot,er cr ptograp,ic c,ecksum tools is sufficient to detect t,ese "ro1an ,orse programs$. Look for signs of a network sniffer. Intruders ma install a network7monitoring program0 commonl called a sniffer 3or packet sniffer60 to capture user account and password information- ",e first step to take in determining if a sniffer is installed on our s stem is to see if an processes currentl ,as an of our network interfaces in promiscuous mode0 t,en a sniffer could be installed on our s stem- Note t,at detecting promiscuous interfaces will not be possible if ou ,ave rebooted our mac,ine or are operating in single user mode since our discover of t,is intrusion- 8eep in mind t,at some legitimate network monitors and protocol anal /ers )

will set a network interface in promiscuous mode- .etecting an interface in promiscuous mode does not necessaril mean t,at an intruderBs sniffer is running on a s stem%. Examine files run !y &cron and &at'. Intruders ma leave back doors in files run from cron or submitted to at- ",e cron and at commands are used to sc,edule commands and processes t,at repeat at specified intervals",ese tec,ni2ues can let an intruder back on t,e s stem 3even after ou believe ou ,ad addressed t,e original compromise6- Also0 verif t,at all files/programs referenced 3directl or indirectl 6 b t,e cron and at 1obs# at l # crontab -l

(. #heck for unauthori)ed services. Inspect /etc/inetd.conf or /etc/xinetd.conf for unaut,ori/ed additions or c,anges- In particular0 searc, for entries t,at e&ecute a s,ell program3for e&ample0 / !in/sh or /!in/csh6 and c,eck all programs t,at are specified in /etc/inetd.conf to verif t,at t,e are correct and ,avenBt been replaced b "ro1an ,orse programs- Also c,eck for legitimate services t,at ou ,ave commented out in our /etc/inetd.conf- Intruders ma turn on a service t,at ou previousl t,oug,t ou ,ad turned off0 or replace t,e inetd program wit, a "ro1an ,orse program*. Examine /etc/passwd file. C,eck t,e /etc/passwd file on t,e s stem for modifications to t,at file0 and look for t,e unaut,ori/ed creation of new accounts0 accounts wit, no passwords0 or UI. c,anges 3especiall UI. ;6 to e&isting accounts- >or e&ample0 we can use stat /etc/passwd to look at access and modification information0 also use cat /etc/passwd to s,ow t,e contents of t,e file+. #heck system and network configuration. >ind an unaut,ori/ed entries0 especiall CDB3plus sign6 entries and inappropriate non7 local ,ost names in /etc/hosts- >urt,ermore0 confirm t,at t,ese files e&isted prior to an intrusion and were not created b t,e intruder,. Look everywhere for unusual or hidden files. >iles t,at start wit, a period . and are normall not s,own b ls can be used to ,ide tools and information- A common tec,ni2ue on UNIX s stems is to put a ,idden director in a userBs account wit, an unusual name0 somet,ing like CEB or C--B or C--F4B- ",e find program can be used to look for ,idden files0 for e&ample?
# find / -name .. print xdev # find / -name . ! print xdev " cat v

1-. Examine all machines on the local network. @ost of t,e time0 if one ,ost ,as been compromised0 ot,ers on t,e network ,ave been0 too3C$<" Coordination Center0 15556

2a(s to !reserve evidence

As soon as ou ,ave evidence t,at indicates a compromise of t,e s stem0 w,ic, cannot be refuted b ot,er evidence ou ,ave at ,and0 ou must assume t,e s stem ,as in fact been compromised and begin to gat,er evidence before it is destro ed0 before logs e&pire0 or before an t,ing is altered 3.ittric,0 );;;6+efore collecting evidence0 some preparations are needed? 1- 8eep a regular old notebook ,and and take careful notes of w,at ou do during our investigation)- Start wit, t,e script command to ensure t,at ever t,ing ou do is captured in a file@ake t,e current time and t,e ,ostname t,e first two items to appear in t,e script fileH script script.txt # date # uname a G- Since s stem binar programs suc, as login, su, telnet, ls, and find could ,ave been altered0 it is better to use our own clean tools C. to read all t,e files t,at ou are going to investigateIou must be logged on as root in order to access our tools C.- @ount our tools C. and set our pat, so t,at t,e C. is t,e onl t,ing in ou #A": variable= t,en confirm our pat,# mount -t iso#$$0 /dev/cdrom /mnt/cdrom # %&'( ) /mnt/cdrom # ec*o +%&'( /mnt/cdrom

!e will know discuss wa s to preserve evidence according to its volatilit and utilit #./ storage a. <egisters? Infeasible to capture and of minimal utilit !. Cac,es? Infeasible to capture as a discrete entit but s,ould be captured as part of s stem memor image 38ruse 9 :eiser0 );;)6-

!,en UNIX s stems are s,utdown wit, t,e shutdown command0 all services are cleanl s,utdown and cac,ed file s stem buffers are flus,ed0 data is written to disc and can be captured as part of s stem memor image0ideo ",e current screen can be captured and provide useful information0 and t,at information is stored in <A@"o save a cop of t,e screen? 1- #ress #trl12lt132 3on a <ed:at Linu& s stem6 to enter anot,er virtual terminal on t,e console and login as root 3if ou know t,e password6)- Use xwd command to dump an individual window or use Jroot option to dump entire screen H xwd displa, local*ost-0 root . screens*ot.xwd G- <eturn to t,e X displa b pressing #trl12lt13* %- $&amine it to make sure t,at ou didnBt capture t,e screen saver wit, xwud command H xwud in screens*ot.xwd 38ruse 9 :eiser0 );;)6 4ystem memory Includes information on all running processes and t,e state of t,e kernel- ",e data is eas to capture0 but t,e act of capturing it c,anges it 38ruse 9 :eiser0 );;)6- UNIX treats ever t,ing as a file0 w,ic, makes it eas to cop and save t,e contents of t,e s stem memor - ",e p, sical memor on a UNIX s stem is / dev/mem= t,e virtual memor is /dev/kmem. !e can use t,e dd command to cop t,e memor from t,e suspect mac,ine?
# dd if )/dev/mem of )suspect.mem.image # dd if )/dev/kmem of ) suspect.kmem.image

",e act of collecting it c,anges it slig,tl = t,erefore0 it is impossible to verif accurac w,en cop ing ,ig,l volatile ob1ects suc, as t,e s stemBs memor 5etwork connections Network state provides important information on bot, current network connections and listening processes- !e need to know w,en an attacker ,as left a network process running or w,en unaut,ori/ed connections are taking place eit,er inbound or outbound 38ruse 9 :eiser0 );;)6",e netstat command can capture information on ongoing network activit and use of t,e Jp option will s,ow t,e processes associated wit, specific network connections# netstat p

It is possible for an attacker to mess wit, t,e routing tables and even ,ack t,e address resolution t,at maps an $t,ernet cardBs @AC addresses to an I# address 38ruse 9 :eiser0 );;)6- !e can use netstat to displa t,e routing tables and arp to capture t,e address resolution tables# netstat rn # arp v

!e can capture all of t,is data in one file along wit, date command to timestamp t,e action# /date0 netstat p0 netstat rn0 arp v1 . suspect.netstatus.txt

6unning processes Uni& supplies a number of utilities t,at provide information on t,e set of all running processes or provide details on a specific running process- ",ose utilities are? ps .ispla s a list of all running processes wit, details about t,eir conte&t and state last, w, who 4et listings of logged in users0 prior logins0 etc uptime S,ows current processing load and ) previous load values- #rovides an understanding of current and recent activit top Useful diagnostic tool w,en s stem is running slowl - #assword cracking tools will s,ow up clearl wit, top lsof #rovides a list of all currentl open files and t,e processes t,at ,ave opened t,em fuser Identifies w,ic, processes are using a specific file or network socket strace List all s stem calls being made b a running process truss, ktrace $arlier versions of s stem call trace ltrace Librar routine trace38ruse 9 :eiser0 );;)6 ",e /proc director is a pseudo7files stem t,at provides a structured interface to /dev/kmem. $ver process in memor ,as a director in /proc associated wit, it named after its process I. 3#I.6- At t,e time of t,is writing0 no attacks t,at ,ide / proc entries ,ave been reported0 making /proc a more reliable guide to currentl running processes t,an t,e commonl ,acked ps utilit K8ruse );;)L Iou s,ould collect a 2uick list of t,e #I. directories in /proc so t,at ou can compare it later wit, t,e output of ps. If a process is missing from t,e ps output t,at does appear in /proc, t,at is a clue t,at ps mig,t ,ave been tro1anned# ls d /proc/20-#3 .suspect4proc.txt

Signs of ,ostile processes? An discrepanc between ps, top, and /proc77#I.s t,at appears in top or /proc but not ps ma ,ave been deliberatel ,idden Unrecogni/ed commands0 especiall ones t,at start wit, a . or ./ are clear indications t,at t,e application was started manuall .aemons running more t,an once t,at s,ould be running onl once0 suc, as inetdM",at means t,e second one was started manuall - ",is is usuall a sign of a tro1ani/ed version-

:ig, uptimes 3,ig, s stem utili/ation6 and especiall a process using an unusual level of s stem resourcesMSome processor intensive program suc, as cr ptanal sis0 bogus I<C servers0 and crack utilities ma be running :ig, network utili/ationMCould be a sign t,at one or more s stems on our network are being used as /ombies in a denial7of7service attack38ruse 9 :eiser0 );;)6

Investigation of a Linux S(ste

",e management of t,e A+C Corporation suspected t,at t,eir production server was t,e target of some malicious activit and re2uested t,e ,elp of t,e forensic investigation team to verif t,e matter- (ne of t,e preconditions imposed on t,e team was t,at t,e production server could not be broug,t down for purpose of investigation- :ence an investigation was e&pected on a live server- Conse2uentl t,e file s stem could not be mounted in t,e read onl mode- :ence @.A c,ecksums of t,e original disks were not made before t,e transfer into t,e evaluation s stem as t,e ver process of taking t,e ,as, value would make it different from t,e value obtained on t,e images at t,e collection s stem- :ence t,e @.A ,as, of t,e suspectBs disk images obtained after t,e images ,ave been completel copied into t,e collection s stem are s,own to be identical to t,e @.A ,as, values of t,eses images after t,e investigation- +ut t,e investigation presumes administrative rig,ts for t,e investigators- ",is is important to run t,e various commands t,at re2uire t,at t,e investigator ,ave root privileges",e investigation we carried out was comprised of two important steps1- ",e collection of evidence for t,e file7s stem anal sis)- ",e outcome of t,e collection process0 and t,e investigation of t,e suspect file7s stem

Co##ection of 3vidence
",e important steps involved in t,e collection p,ase are mentioned ,ere7under1. /sing the &netcat' command. ",e netcat command is used to facilitate cop ing of data across t,e LAN- Netcat ,as to be manuall started on t,e collection ,ost and t,e suspect ,ost- Initiall t,e listening process is started on t,e data collection #C- An arbitar ,ig, numbered port is c,osen suc, as 1;;;;Command on t,e collection mac,ine?
# nc l p 50000 . /temp/nc.suspect.passwd4file

'

Command on t,e suspect mac,ine?

# cat /etc/passwd /etc/s*adow " nc 5#2.5$6.0.2 50000 -w7

",e nc e&ecutable running on t,e listening ,ost runs until it receives a connection and t,en t,e connection is broken- At t,is point t,e e&ecution stops and t,e output file is closed- ",e 7wG option on t,e transmitting ,ost means to wait G seconds after t,e data ,as been sent and t,en timeout2. Log the investigation steps carried out on the suspect system. Command?
# script investigation.txt

",e script command ensures t,at ever t,ing done on t,e suspect s stem console is documented into a file 3investigation-t&t6 for evidentiar value- ",is p,ase is important to carr out t,e investigation as met,odicall and carefull as possible- ",e file would enumerate t,e s stematic flow of commands used on t,e suspect s stem to elicit pertinent information- ",is is also important to convince t,e management for a need for response capabilit if a securit breac, is discovered from t,e investigation". 7aintain a 8ournal to enumerate the results of investigation. ",e findings from t,e commands run or t,e steps taken in t,e investigation are clearl documented so as to s,ow t,e logical development of t,e investigative process and to function as an aid for refres,ing t,e memor and making t,e investigative report$. 0erify the date and time of the suspect system and determine the 954 name and :4 version of the system. Commands?
# date8 # uname a

Commands t,at make t,e current time and t,e ,ostname t,e first two items to appear in t,e script file in t,e suspect s stem- ",e s stem time is furt,er noted in t,e 1ournal maintained b an investigator- An discrepancies between t,e s stem clock and t,e real time would ,ave been noted- ",e second command would also give information about t,e operating s stem for instance t,e version of t,e kernel for t,e particular Linu& (S%. :!tain the screenshot of the suspect system's desktop. Command?
# xwd displa, local*ost-0 root . screens*ot.xwd

",e screens,ot of t,e suspect s stem desktop is obtained wit, t,e N&wdO command and cop is sent to t,e collection s stem using t,e netcat command- A program suc, as t,e

Linu& grap,ics editor 4I@# will ,ave to be used to convert t,e file from t,e &wd format to a more usable format suc, as 1peg-

(. 4ave the copies of the suspect's password and shadow files to the collection1system. Command?
# cat /etc/passwd/etc/s*adow " nc 549.576.$#.:# 50005 w7

",e password and s,adow files are saved and t,e copies are sent to t,e collection s stem using t,e netcat command*. :!tain the copies of the suspect's normal memory and kernel memory files to the collection1system. Command?
# dd bs)5024 ; /dev/mem " nc 549.576.$#.:# 50002 w7 # dd bs)5024 ; /dev/kmem " nc 549.576.$#.:# 50007 w7

Since t,e Linu& considers ever t,ing as a file it makes it eas to cop and save t,e contents of t,e s stem memor +. 6un the command that sends the date, details of processes with network connections, the kernel routing ta!le and the address resolution protocol to the collection system. Command?
# /date0 netstat p0 netstat rn0 arp v1 " nc 549.576.$#.:# 50004 -w7

",e network state provides important information on bot, t,e current network connections and t,e listening processes- ",e commands ,elp t,e investigators to know about possible running processes left b a ,acker or gives details about an unaut,ori/ed connections taking place- ",e NdateO command records t,e date w,en t,e commands were run in t,e script file - ",e Nnetstat JpO command gives t,e processes associated wit, t,e different network connections- ",e Nnetstat JrnO command would s,ow t,e kernel routing table- ",e Carp JvO would give an alternate view of t,e routing table-",is information is copied to t,e collection s stem,. Looking for unusual running processes. A series of steps are taken to use some utilities enumerated below0 to provide valuable information about all running processes- ",e task of t,e investigation team is to capture t,e state of t,e suspect mac,ine and get a list of all open files- ",en t,e team tries to correlate t,is information wit, t,e Cverbose netstatB captured earlieri6 <un t,e command to verif t,e s stems uptime-

Command?
# uptime

",e command finds ,ow muc, time t,e s stem ,as been running after its last reboot",e idea is to c,eck to find evidence of a s stem compromise as t,e ,acker mig,t re2uire a reboot to start some processes- $lse it mig,t also point to a denial7of7service attack as t,e s stem mig,t ,ave cras,ed in panicii6 Information about w,o is connected to s stem remotel Command?
# w*o

",e command c,ecks t,e users logged in to t,e s stem via t,e telnet or t,e ftp servicesiii6 C,eck on t,e process t,at takes t,e ma&imum resources of t,e s stem Command?
# top

",e command provides t,e real time displa of t,e C#U7intensive processes- It is a diagnostic tool w,en t,e s stem is running slowl - #assword cracking tools will s,ow clearl wit, t,e CtopB command- If t,e team detects an process utili/ing t,e C#U resources in an out7of7t,e7ordinar fas,ion t,at can potentiall point to a s stem securit breac, t,en command like CstraceB or CltraceB or CfuserB would be run",e strace command? It places a S stem Call "race- It lists all s stem calls being made b a running process",e ltrace command? It places a librar routine trace",e fuser command ? It is a file user command t,at identifies w,ic, process is using a specific file or network socket1-. :!tain information on the running processes and their details along with a list of open files and send it to the collection system. Command?
# /ps aux0 ps auxeww0 lsof1 " nc 549.576.$#.:# 5000: -w7

",e first part of t,e command provides information about t,e processes t,at is convenient to be printable on a standard screen- ",e second part gives detailed information of t,e processes for later anal sis- ",e last part of t,e command gives t,e list of open files11. 3ind the list of running processes on the suspect system. Command?
# ls d /proc/20-#3 " nc 549.576.$#.:# 5000$ -w7

1;

",e commands give a director listing of t,e contents of t,e proc director - It is wort, to note t,at w,en a kernel is ,acked0 even an unmodified cop of ps can provide misleading information- ",e process names in t,e ps output can be c,anged wit,out an modification of s stem binaries12. 7ake a tar file of the /proc directory and send it to the collection system. Command?
# tar cvpf proc.tar /proc/20-#3

",e CtarB command is used to create an arc,ive- ",e parameters used are CcB to create a new arc,ive0 CvB to be in t,e verbose mode to provide a complete list of w,at t,e process is doing0 CpB to preserve owners,ip so t,at if t,e file is tampered wit, or looks out7of7 ordinar it could be pointed out b wa of owners,ip0 and CfB to send t,e output to a file1". 9etermine the physical devices in the suspect system. Command?
# mount

",is is t,e first step to cop t,e entire file7s stem over t,e network to t,e collection s stem- ",e CmountB command ,elps t,e team to determine t,e active file7s stems on t,e ,ard drive- It would also point to an network file s stems t,at are being used1$. 9etermine the partitions on the primary hard drive of the suspect system. Command? H fdisk l /dev/*da ",e utilit is used as t,e disk partition utilit t,at ,elps t,e team to find t,e list of partitions on t,e primar ,ard drive and also find t,e file7s stems on t,e ,ard drive1%. 7ake a data dump of the partitions of the suspect system and send it to the collection system. Command?
# dd if)/dev/*da5 bs)5024 " nc 549.576.$#.:# 50009 -w7

",e data dump command ,elps to collect eac, image of t,e partitions of t,e ,ard disk and send it to t,e collection s stem1(. 7ake the 79% hash on all the files received and the image of the hard1disk partitionsCommand?
# md:sum b suspect.*da5.image

11

",e @.A ,as, of t,e images of t,e partitions is taken before investigation- After t,e compete investigation t,e @.A is taken again to s,ow t,at t,e investigation did not tamper an of t,e images-

0na#(sis of 3vidence
",e utmost concern from t,e start of t,e investigation was t,e fact t,at t,e suspect s stem was a live server0 and t,at its normal operation could not be ,alted in order to perform t,e investigation- ",at fact limited our abilit to create e&act copies of t,e file s stem- Since t,e s stem could not be fro/en in time0 t,e creditabilit of t,e investigation ,inged on t,e abilit to collect as muc, information as possible in a s,ort period of time0 and t,en ensure t,e integrit of t,e information once it was collectedIn order to ensure t,at t,e collection p,ase went as smoot,l as possible0 t,e investigating team first sat down and developed t,e investigation c,ecklist t,at ,as alread been outlined- ",e investigation team consisted of two people- ",e first person was responsible for p, sicall interacting wit, t,e s stem console0 and t,e second person was responsible for maintaining t,e c,ecklist and t,e p, sical log of t,e investigation",e suspect s stem was a .ell (ptiple& 4X1 desktop computer running <ed:at Linu& version P-;- ",e investigation team ,ad t,e fortune of ,aving access to a .ell (ptiple& 4X11; desktop computer t,at was also running <ed:at Linu& version P-; t,at could be used as t,e collection s stem- +ot, s stems were running version )-%-1P71% of t,e Linu& kernel- In order to be certain of t,e integrit of t,e collection s stem0 t,e computer was formatted and a fres, install of Linu& was placed on t,e s stem- ",e install media was created from disk images downloaded from a <ed:at mirror site- #rior to burning t,e disk images to C.7<(@0 @.A sums were taken of t,e disk images and t,ose sums were verified against t,e sums stored on t,e main <ed:at download site- (nl software packages present on t,e disk media were installed on t,e collection s stem(nce t,e c,ecklist was prepared0 and t,e collection s stem was readied0 t,e collection p,ase of t,e investigation began- !,en t,e team approac,ed t,e s stem it was noted t,at t,e s stem was on0 and t,at t,e monitor was in power save mode- @oving t,e mouse broug,t up t,e desktop- ",ere were no windows visibl open on t,e desktop- ",e investigators opened a single terminal window in order to perform t,e data collection- ",e root user was logged into t,e console- (n t,e suspect s stem t,e script command was invoked beginning evidence collection process0 and on t,e collection s stem a folder was created t,at would ,old t,e findings of t,e investigation",e collection process followed t,e c,ecklist wit,out error until t,e step w,ere t,e investigators attempted to cop t,e s stemBs kernel memor 3/ dev/kmem6- !,en t,e data dump of t,e kernel memor was attempted t,e s stem returned t,e following error message-

1)

H dd bsQ1;)% R /dev/kmem S nc 1%'-1GP-*5-*5 1;;;G 7w G dd? reading Tstandard inputU? +ad address ;D; records in ;D; records out ",e data file on t,e collection s stem was empt - ",e command was attempted several times wit,out success on t,e suspect s stem- ",e investigators also modified t,e s nta& of t,e command during t,e subse2uent attempts to see if t,ere ,ad been an error in t,e original s nta&Instead of reading t,e /dev/kmem file in using t,e input c,aracter NRO0 as t,e 8ruse 9 :eiser te&t did in some instances0 we also tried t,e input file argument0 NifQO0 t,at is part of t,e dd command- !e did not ,ave success wit, eit,er version- >ollowing t,e investigation0 t,e original s nta& of t,e command was tested on t,e collection s stem and two ot,er Linu& s stems- ",e data dump failed in t,e same manner on t,e collection s stem as it did on t,e suspect s stem",e first alternate Linu& s stem was running <ed,at Linu& version *-) wit, version )-)-1%7A-; of t,e Linu& kernel0 and t,e second alternate Linu& s stem was running an earl release of Slackware Linu& wit, version )-;-; of t,e Linu& kernel- ",e data dump worked correctl on t,ose alternate s stems- No reason to date ,as been found to e&plain w, t,e data dump of t,e kernel memor failed on t,e two newer s stems>ollowing t,e set back of not being able to collect t,e kernel memor 0 t,e investigation team continued wit, t,e c,ecklist- ",e output of t,e uptime0 who0 and top commands did not uncover an activit t,at appeared to be out of t,e ordinar 0 so it was decided b t,e investigation team to forego an e&,austive documentation of t,e running processes using t,e strace0 ltrace0 and fuser commands- ",e final step of t,e data collection p,ase was to collect t,e images of t,e partitions on t,e s stemBs ,ard drives- ",e mount and fdisk commands were used to determine t,e number of ,ard drives present on t,e s stem0 and ,ow man partitions t,e were divided into- (ne ,ard drive was present on t,e suspect s stem0 and t,at one drive was divided into four partitions- ",ree of t,e four partitions were used b t,e regular file7s stem0 and t,e fourt, was used as t,e swap space for t,e s stem- All of t,e partitions on t,e ,ard drive were collected using t,e data dump command- ",e data dump command appeared to operate normall 0 but at t,e end of eac, operation t,e following error message appeareddd? reading T/dev/,da1U? Input/output error It is our opinion t,at t,is error resulted from t,e fact t,at t,e partitions were mounted in read7 write mode t,roug,out t,e investigation process>ollowing t,e creation of t,e disk images0 t,e investigation team attempted to mount t,e suspect file s stems on t,e collection s stem- Attempts to mount t,e images on t,e collection s stem in a read7onl mode resulted in an error message stating t,at t,e superblock for t,e image appeared to be damaged- ",e investigators t,en mounted t,e image for /dev/hda1 in normal read7write mode0 and t,e image t,en mounted normall - ",is process corrected t,e error in t,e superblock0 but it also c,anged t,e contents of t,e image file t,us invalidating an @.A sum taken prior to t,e mount process- In order to ensure t,e integrit of t,e ot,er disk images @.A sums were created0 and t,e file s stems were not mounted during t,e anal sis p,ase of t,e investigation- ",e t,ree unmounted images represented t,e ,ome directories for t,e suspect

1G

s stem0 t,e swap space0 and t,e /usr director - After mounting t,e suspectBs root file7s stem in read7write mode0 t,e image was immediatel unmounted0 and remounted in read7onl mode.eeper investigation of t,e image uncovered ot,er files t,at were damaged during t,e data dump- An files t,at were written to t,e ,ard drive during t,e data dump suc, as log files were empt - ",e file names were present0 but t,e contents of t,e files were missing- ",is proved to be a ma1or set back to our effort- !e were forced to immediatel return to t,e suspect s stem and make a tar ball of t,e main log director 0 /var/log0 and t,en cop t,at tar ball to t,e collection s stem- !e were also forced to perform our anal sis of t,e files present in t,e /home0 and /usr directories on t,e live suspect s stem0 instead of anal /ing t,e images we created>ollowing t,e collection p,ase t,e investigation s,ifted focus to t,e collection s stem and we began to process t,e data t,at we ,ad collected- ",e first item of data anal /ed was t,e concatenated file t,at we ,ad created t,at contained bot, t,e /etc/passwd and /etc/shadow files",ere were not an suspicious or unaut,ori/ed entries in eit,er file- ",e ne&t item of data anal /ed was t,e file containing t,e output from t,e two netstat commands and t,e arp command- ",e onl active network connection reported b t,e set of commands was t,e netcat process t,at was cop ing t,e data to t,e collection s stem- At t,is point we also looked at t,e output from t,e two ps commands0 and t,e lsof command- As wit, t,e network statistics0 we did not find an processes t,at appeared to be out of place- In order to be certain t,at t,e suspectBs ps command was not tampered wit, b an intruder we also investigated t,e /proc director directl - ",e investigation did not find an processes t,at were not reported b t,e ps command+ t,is point in time0 it was becoming apparent t,at if t,e s stem ,ad been attacked0 t,e attack ,ad b t,is time ceased- Now t,at it seemed t,at we were dealing wit, a ,istorical event we turned our attention to t,e log files t,at we collected from t,e suspect s stem",e first set of logs to be anal /ed were t,e web server logs0 since t,is serverBs primar function was serving web pages- Since t,e access and error logs for t,e server were relativel s,ort we processed t,e log manuall - ",e page accesses were normal0 and did not include an evidence of script7kiddie attacks",e second log anal /ed was t,e messages log in t,e /var/log director 0 and t,is log proved to be most ,elpful- .uring a time spanning from appro&imatel 1)?1' #@ $S" to )?G1 #@ $S" on >rida .ecember *0 );;) we see t,at two separate unaut,ori/ed ,osts attempted to gain access to t,e suspect s stem- ",e first ,ost0 venger-bridgewater-edu 0 focused ,is attack solel on t,e ftp service running on t,e s stem- ",e ,ost made twelve attempts to log into t,e s stem as eit,er t,e root user0 or as an unprivileged user named 1oe- ",is attack appears to ,ave been performed manuall b a user on t,e unaut,ori/ed ,ost because of t,e relativel low number of attempts and t,e amount of time over w,ic, t,e attacks occurred- +elow is an e&cerpt of t,e failed attempts.ec * 1)?AG?1; dorm*57A) ftp3pamVuni&6K1*1GL? aut,entication failure= lognameQuidQ; euidQ; tt Q/dev/ftpd1*1G ruserQ r,ostQWenger-bridgewater-edu userQ1oe .ec * 1)?AG?1) dorm*57A) ftpd? Wenger-bridgewater-edu? connected? I.L$K1*1GL? failed login from Wenger-bridgewater-edu K1%'-1GP-);-1AL .ec * 1%?1%?)) dorm*57A) ftp3pamVuni&6K15G%L? aut,entication failure= lognameQuidQ; euidQ; tt Q/dev/ftpd15G% ruserQ r,ostQWenger-bridgewater-edu userQ1oe

1%

.ec * 1%?1%?)A dorm*57A) ftpd? Wenger-bridgewater-edu? connected? I.L$K15G%L? failed login from Wenger-bridgewater-edu K1%'-1GP-);-1AL ",e second ,ost0 saturn-bridgewater-edu0 did not attempt to access t,e s stem until 1?1' #@ $S"- ",e second ,ost initiall attempted to open an anon mous ftp session wit, t,e suspect s stem- ",is attempt failed because t,e s stem was not running an anon mous ftp service- ",e second ,ost did not attempt to access t,e s stem again until )?)' #@ $S"- At t,at time t,e attacker attempted to remotel login to t,e s stem eit,er as t,e root user0 t,e lp user0 or as a blank username using t,e telnet service- +elow is an e&cerpt of t,e attackerBs attempts.ec * 1%?)'?;1 dorm*57A) login3pamVuni&6K))AGL? aut,entication failure= lognameQ uidQ; euidQ; tt Qpts/; ruserQ r,ostQSaturn userQlp .ec * 1%?)'?;G dorm*57A) loginK))AGL? >AIL$. L(4IN 1 ><(@ Saturn >(< lp0 Aut,entication failure .ec * 1%?)'?;P dorm*57A) login3pamVuni&6K))AGL? bad username KL .ec * 1%?)'?1; dorm*57A) loginK))AGL? >AIL$. L(4IN ) ><(@ Saturn >(< 0 Aut,entication failure .ec * 1%?)'?1* dorm*57A) login3pamVuni&6K))AGL? aut,entication failure= lognameQ uidQ; euidQ; tt Qpts/; ruserQ r,ostQSaturn userQroot .ec * 1%?)'?15 dorm*57A) loginK))AGL? >AIL$. L(4IN G ><(@ Saturn >(< root0 Aut,entication failure >our minutes after t,is attempt0 t,e second attacker appears to ,ave performed a port scan of t,e suspect s stem- ",e attacker did not tr to ,ide t,e attempt0 because t,e scan appears to ,ave been a full "C# connect3 6 scan0 and t,e most likel tool used would ,ave been nmap- ",e finger0 telnet0 and ftp services all logged connection attempts in t,e message file- +elow is an e&cerpt from t,at probable port scan.ec * 1%?G1?;' dorm*57A) fingerdK))'GL? Client ,ung up 7 probable port7scan .ec * 1%?G1?11 dorm*57A) &inetdK))'%L? CanUt get t,e number of pending signals? +ad file descriptor 3errno Q 56 .ec * 1%?G1?11 dorm*57A) last message repeated A1 times .ec * 1%?G1?)% dorm*57A) ftpdK))'*L? wu7ftpd 7 "LS settings? control allow0 cli entVcert allow0 data allow .ec * 1%?G1?)% dorm*57A) ftpdK))'*L? lost connection to Saturn-bridgewater-edu K1%'-1GP-1;-%;L .ec * 1%?G1?)% dorm*57A) ftpdK))'*L? >"# session closed .ec * 1%?G1?)A dorm*57A) telnetdK))''L? ttloop? read? Connection reset b peer >ollowing t,is port scan connection attempts from bot, of t,e unaut,ori/ed s stems stopped!e now ,ad evidence t,at at least one individual attempted to gain unaut,ori/ed access to t,e suspect s stem- (ur efforts now focused on determining w,et,er or not t,e an traces of successful entr e&isted on t,e suspect s stem- !e c,ecked t,e xferlog for ftp file transfers during t,e time in 2uestion0 and found not,ing- ",e ne&t step was to determine if t,ere were an unaut,ori/ed trust relations,ips in t,e /etc/hosts file- ",ere were no trust relations,ips on t,e suspect s stem at all- !e ne&t searc,ed t,roug, t,e /dev director to ensure t,at no regular files

1A

were being ,idden t,ere0 and none were found- !e also searc,ed for e&ecutable files and suspicious ,idden files in t,e userBs ,ome directories0 and again not,ing turned up out of t,e ordinar - Ne&t we c,ecked for unusual SUI. and S4I. files0 and not,ing appeared out of t,e ordinar - ",e final searc, of t,e file s stem was for core files- If t,e attackers ,ad been able to panic an of t,e network services during t,eir attempts0 core dumps of t,e processes s,ould ,ave e&isted on t,e s stem- ",e searc, did not turn up an core files+ t,is point we were confident t,at t,e attackers ,ad not found a wa into t,e s stem0 but we were not going to take an t,ing for granted- ",e final step in t,e investigation was to run t,e rpm 10a command- ",is command is used to verif t,e consistenc of t,e binar packages installed on t,e suspect s stem using t,e <ed,at #ackage @anager- !,en 2uestioned t,e administrator ,ad stated t,at all of t,e binar packages present on t,e s stem ,ad been installed using t,e package manager- ",erefore0 we could determine if an of t,e services running on t,e s stem could ,ave been tro1anned- ",e output of t,e command s,owed t,at t,e ke programs on t,e s stem ,ad not been replaced- ",e onl files t,at were reported b t,e command to ,ave c,anged since t,e install of t,e packages were log files0 and s stem configuration files t,at were updated b t,e normal operation of t,e s stem",e investigation of t,e suspect s stem was now complete- $vidence was found to s,ow t,at attackers ,ad tried to gain unaut,ori/ed access to t,e s stem0 but t,ose attackers did not gain access to t,e suspect s stem- ",e administrators of t,e s stem were informed of t,e completion of t,e investigation0 and t,e findings of t,e investigation-

Su

ar(

As t,is document ,as ,opefull s,own0 t,e most important part of investigating a UNIX or Linu& s stem is ,aving a defined procedure to follow- ",ese s stems contain a wealt, of information if ou know w,ere important information is stored0 and ,ow to retrieve t,at information- ",e tools needed to perform a basic intrusion detection anal sis are alread on t,e s stem- !e did not use an tools or utilities t,at were not included on t,e <ed,at installation disks- It s,ould also be noted t,at t,e 8ruse 9 :eiser te&t provided a wonderful blueprint for ,ow to gat,er t,e pertinent evidence from t,e s stem- In some cases we ,ad to modif commands to suit our needs0 or correct for s nta& or logic errors0 but overall t,e te&t was our most valuable resource during t,e preparation of t,is documentIn t,e end0 we were able to c,eapl and 2uickl perform an investigation0 and come to a conclusion about t,e state of t,e suspect s stem- ",e investigation was not detrimental to t,e suspect s stem in an wa - It was able to continue to perform its duties t,roug,out t,e investigation0 and we left t,e s stem in a full operational state- ",e investigation was trul a success-

1*

'i/#iogra!*(
C$<" Coordination Center 315556- Intruder Detection Checklist. U<L? ,ttp?//www-cert-mil/tec,tips/intruderVdetectionVc,ecklist-,tm C$<" Coordination Center 315556- Steps for recovering from a UNIX Root Compromise. U<L? ,ttp?//www-cert-mil/tec,tips/rootVcompromise-,tm .ittric,0 .avid 3);;;6- N+asic Steps in >orensic Anal sis of Uni& S stemO- U<L? ,ttp?//staff-was,ington-edu/dittric,/talks/black,at/black,at/forensic-,tml 4ollmann0 .ieter 315556- Computer Security. New Iork0 NI? Xo,n !ile 9 Sons LtdYA'*-5-A)A4*A 1555= IS+N ;7%'175'P%%7)8ruse0 !arren 4-0 9 :eiser0 Xa 4- 3);;)6- Computer Forensics Incident Response !ssentials. New Iork0 NI? Addison !esle - YA'*-5-A)A 8'P );;1= IS+N ;7);17';'157A"obler0 @ic,ael X- 3);;16- Inside "inu#. Indianapolis0 IN? New <idersYA'*-'*-(*G "A5P );;1= IS+N ;7'GA'7;5%;7P Wolkerding0 #-0 <eic,ard0 8-0 9 Xo,nson0 $ 3155*6- "inu# configuarion $ installation. New Iork% NI? @IS #ress- YA'*-'*-(*G W*% 155*= IS+N 17AAP)P7%5)7G

1'