Some OS Basics

SomenotesadoptedfromBryantandOHallaron

Role of The Operating System?

Processes
Def:Aprocessisaninstanceofarunningprogram.
Oneofthemostprofoundideasincomputerscience. Notthesameasprogramorprocessor

Processprovideseachprogramwithtwokeyabstractions:
Logical i lcontrol lfl flow
EachprogramseemstohaveexclusiveuseoftheCPU.

Privateaddressspace
Eachprogramseemstohaveexclusiveuseofmainmemory.

HowaretheseIllusionsmaintained?
Processexecutionsinterleaved(multitasking) Addressspacesmanagedbyvirtualmemorysystem

Logical Control Flows

Each process has its own logical control flow
Process A Time Process B Process C

Concurrent Processes
Twoprocessesrunconcurrently (areconcurrent) if theirflowsoverlapintime. Otherwise,theyaresequential. Examples:
Concurrent:A&B,A&C Sequential:B&C
Process A Time Process B Process C

User View of Concurrent Processes

Controlflowsforconcurrentprocessesare physicallydisjointintime. However,wecanthinkofconcurrentprocessesare ginp parallelwitheachother. running
Process A Time Process B Process C

Context Switching
ProcessesaremanagedbyasharedchunkofOScode calledthekernel Controlflowpassesfromoneprocesstoanotherviaa contextswitch.

Process A code

Process B code
user code context switch

Time

k kernel l code d user code kernel code user code

context switch

Process: Traditional View

Process=processcontext+code,data,andstack
Processcontext
Programcontext: D registers Data i Conditioncodes Stackpointer(SP) Programcounter(PC) Kernelcontext: VMstructures Descriptor p table brk pointer SP Code,data,andstack stack sharedlibraries brk PC 0 runtimeheap read/writedata readonlycode/data

Process: Alternative View

Process=thread+code,data,andkernelcontext
Thread
Programcontext: g Dataregisters Conditioncodes Stackpointer(SP) Programcounter(PC) bk brk PC 0 SP stack Kernelcontext: VMstructures Descriptortable brk pointer Code,data,andkernelcontext sharedlibraries runtimeheap read/writedata readonlycode/data

Process with Two Threads

Thread1
Programcontext: D t registers Data it Conditioncodes Stackpointer(SP) Programcounter(PC) stack C d d Code, data, t and dkernel k lcontext t t sharedlibraries brk PC 0 Kernelcontext: VMstructures Descriptortable brk pointer runtimeheap read/writedata readonlycode/data

SP

Thread2
Programcontext: Dataregisters Conditioncodes Stackpointer(SP) Programcounter(PC) stack

SP

Threads vs. Processes

Threadsandprocesses:similarities
Eachhasitsownlogicalcontrolflow Eachcanrunconcurrentlywithothers Eachiscontextswitched( (scheduled) )by ythekernel

Threadsandprocesses:differences
Threadssharecodeanddata data,processes(typically)do not Threadsaremuchlessexpensivethanprocesses
Processcontrol(creatingandreaping)ismoreexpensiveas threadcontrol Contextswitchesforprocessesmuchmoreexpensivethanfor threads

Threads vs. Processes (contd.)

Processesformatreehierarchy Threadsformap poolofp peers
Eachthreadcankillanyother Eachthreadcanwaitforanyotherthreadtoterminate Mainthread:firstthreadtoruninaprocess
Processhierarchy P0 T1 P1 sh sh foo sh T5 T3 sharedcode,data andkernelcontext Threadpool T2 T4

Virtual Memory (Previous Lectures)

Programsrefertovirtualmemoryaddresses

000 movl (%ecx),%eax Conceptuallyverylargearrayofbytes Eachbytehasitsownaddress Actuallyimplementedwithhierarchyofdifferentmemorytypes Systemprovidesaddressspaceprivatetoparticularprocess

Allocation:Compilerandruntimesystem
Wheredifferentprogramobjectsshouldbestored Allallocationwithinsingle g virtualaddressspace p

Butwhyvirtualmemory? Whynotphysicalmemory?

FFF

Problem 1: How Does Everything Fit?

64bitaddresses: 16E Exabyte b t Physicalmainmemory: F Gigabytes Few Gi b t

Andtherearemanyprocesses.

Problem 2: Memory Management

Physicalmainmemory

Process1 P Process2 Process3 P Process n

stack heap

.text .data data

Whatgoes where?

Problem 3: How To Protect

Physicalmainmemory Processi

Processj

Problem 4: How To Share?

Physicalmainmemory Processi

Processj

Solution: Level Of Indirection

Eachprocessgetsitsownprivatememoryspace Solvesthep previousp problems
Virtualmemory

Process1
Physicalmemory

mapping
Virtual i lmemory

Processn

Address Spaces
Virtualaddressspace
SetofN=2nvirtualaddresses:{0,1,2,3,,N1}

Physicaladdressspace
SetofM=2mphysicaladdresses:{0,1,2,3,,M1}

Cleandistinctionbetweendata(bytes)andtheir attributes(addresses) Eachobjectcannowhavemultipleaddresses Everybyteinmainmemory: onephysicaladdress,one(ormore)virtual addresses

A System Using Physical Addressing

Mainmemory 0: 1: 2: 3: 4: 5: 6: 7: 8:

CPU

Physicaladdress (PA)

... .
M1:
Dataword

Usedinsimplesystemslikeembedded microcontrollersindeviceslikecars,elevators,and digitalpictureframes

A System Using Virtual Addressing

CPUChip
CPU
Virtualaddress (VA) Physicaladdress (PA)

MMU

Mainmemory 0: 1: 2 2: 3: 4: 5: 6: 7: 8:

...
M1:
Dataword

Usedinallmoderndesktops,laptops,workstations Oneofthegreatideasincomputerscience MMUchecksthecache

Why Virtual Memory (VM)?

Efficientuseoflimitedmainmemory(RAM)
UseRAMasacacheforthepartsofavirtualaddressspace
somenoncached h dpartsstored dond disk k some(unallocated)noncachedpartsstorednowhere

Keep ponly yactiveareasofvirtualaddressspace p inmemory y

transferdatabackandforthasneeded

Simplifiesmemorymanagementforprogrammers
Eachprocessgetsthesamefull,privatelinearaddressspace

Isolatesaddressspaces
Oneprocesscantinterferewithanothersmemory
becausetheyoperateindifferentaddressspaces

Userprocesscannotaccessprivilegedinformation
differentsectionsofaddressspaceshavedifferentpermissions

Address Translation: Page Tables

Apagetableisanarrayofpagetableentries(PTEs)that mapsvirtualpagestophysicalpages.Here:8VPs
PerprocesskerneldatastructureinDRAM
Physicalpage numberor Valid diskaddress PTE0 0 null
1 1 0 1 0 0 PTE7 1

Physicalmemory (DRAM)
VP1 VP2 VP7 VP4

PP0 PP3

null

Virtualmemory (disk)
VP1 VP2 VP3 VP4 VP6 VP7

Memoryresident pagetable (DRAM)

Address Translation With a Page Table

Pagetable baseregister (PTBR)

Virtualaddress
Virtualpagenumber(VPN) Virtualpageoffset(VPO)

Page g tableaddress forprocess

Pagetable
Valid Physicalpagenumber(PPN)

Validbit=0: pagenotinmemory (pagefault)

Physicalpagenumber(PPN)

Physicalpageoffset(PPO)

Physicaladdress

Exceptions

SomenotesadoptedfromBryantandOHallaron

Control Flow
ComputersdoOnlyOneThing
Fromstartuptoshutdown,aCPUsimplyreadsand executes(interprets)asequenceofinstructions Thissequenceisthesystemsphysicalcontrolflow (or flowofcontrol).
Physical control flow Ti Time <startup> inst1 inst2 inst3 instn <shutdown>

Altering the Control Flow

UptoNow:twomechanismsforchangingcontrolflow:
Jumpsandbranches Callandreturnusingthestackdiscipline. Bothreacttochangesinprogramstate.

Insufficient I ffi i tfor f auseful f lsystem t

DifficultfortheCPUtoreacttochangesinsystemstate.
dataarrivesfromadiskoranetworkadapter adapter. Instructiondividesbyzero Userhitsctlcatthekeyboard S System timer i expires i

Systemneedsmechanismsforexceptionalcontrolflow

Exceptional Control Flow

Mechanismsforexceptionalcontrolflowexistsatalllevels ofacomputersystem. LowlevelMechanism
exceptions
change h i incontrol t lflow fl in i responseto t asystem t event t(i (i.e.,change h i in systemstate)

CombinationofhardwareandOSsoftware

HigherLevelMechanisms
Processcontextswitch Signals Nonlocaljumps(setjmp/longjmp) I l Implemented dby b either: ih
OSsoftware(contextswitchandsignals). Clanguageruntimelibrary:nonlocaljumps.

System context for exceptions

Keyboard Mouse Modem Printer

Processor

Interrupt controller t ll

Keyboard controller t ll

Serial port controller t ll

Parallel port controller t ll

Local/IO Bus

Memory

IDE disk controller

SCSI controller SCSI bus

Video adapter p

Network adapter p

disk disk CDROM

Display

Network

Exceptions
AnexceptionisatransferofcontroltotheOSinresponse tosomeevent(i.e.,changeinprocessorstate)
UserProcess exception exceptionprocessing b exception by ti h handler dl returntoI_current returntoI_next abort OS

event

I_current I_next

Examples: p divby0,arithmeticoverflow,pagefault,I/Orequest completes,CtrlC

Interrupt Vectors
Exception numbers b code for exception handler 0 interrupt vector
0 1 2 n-1

code for exception handler 1 code for exception handler 2

...

...
code for exception handler n-1

Eachtypeofeventhas auniqueexception numberk Indexintojumptable ( k interrupt (a.k.a., i vector) Jump J t table bl entry t k pointstoafunction (exceptionhandler). handler) Handlerkiscalledeach timeexceptionk occurs.

Asynchronous Exceptions (Interrupts)

Causedbyeventsexternaltotheprocessor
Indicatedbysettingtheprocessorsinterruptpin handlerreturnstonextinstruction.

Examples:
I/Ointerrupts
hittingctlcatthekeyboard arrivalofapacketfromanetwork arrivalofadatasectorfromadisk

Hardresetinterrupt
hittingtheresetbutton

Softresetinterrupt
hittingctlaltdeleteonaPC

A Typical Hardware System

CPU chip register file ALU system bus memory bus

bus interface

I/O bridge

main memory

I/O bus USB controller mouse keyboard graphics adapter monitor disk disk controller

Expansion p slots for other devices such as network adapters.

Reading a Disk Sector: Step 1

CPU chip register file ALU

CPU initiates a disk read by writing a command, logical block number, and destination memory y address to a port (address) associated with disk controller.

bus interface

main memory

I/O bus

USB controller mouse keyboard

graphics adapter monitor

disk controller

disk

Reading a Disk Sector: Step 2

CPU chip register file ALU

Disk controller reads the sector and performs a direct memory access (DMA) transfer into main memory. y

bus interface

main memory

I/O bus

USB controller mouse keyboard

graphics adapter monitor

disk controller

disk

Reading a Disk Sector: Step 3

CPU chip register file ALU

When the DMA transfer completes, the disk controller notifies the CPU with an interrupt ( (i.e., asserts a special interrupt pin on the CPU)

bus interface

main memory

I/O bus

USB controller mouse keyboard

graphics adapter monitor

disk controller

disk

Synchronous Exceptions
Causedbyeventsthatoccurasaresultofexecutingan instruction:
Traps
Intentional Examples: p system y calls, ,breakpoint p traps, p ,special p instructions Returnscontroltonextinstruction

Faults
Unintentional lbut b possibly bl recoverable bl Examples:pagefaults(recoverable),protectionfaults(unrecoverable). Eitherreexecutesfaulting(current)instructionoraborts.

Aborts
unintentionalandunrecoverable Examples: Examples parityerror error,machinecheck check. Abortscurrentprogram

Trap Example
OpeningaFile
Usercallsopen(filename,options)
0804d070 <__libc_open>: . . . 804d082: cd 80 804d084: 5b . . . int pop $0x80 %ebx

Functionopenexecutessystemcallinstructionint

OSmustfindorcreatefile,getitreadyforreadingor writing Returns R integer i file fil descriptor d i

User Process
int pop

OS

exception Open file return

Fault Example #1
MemoryReference
Userwritestomemorylocation Thatportion(page)ofusersmemoryiscurrentlyondisk Page g handlermustloadpage p g intop physical y memory y int a[1000]; Returnstofaultinginstruction main () { Successful Success u o onseco second d t try y
a[500] = 13; }

User Process

OS

event

movl

page fault return page g and load Create p into memory

Fault Example #2
MemoryReference
Userwritestomemorylocation int a[1000]; main () Addressisnotvalid { a[5000] = } Page g handlerdetectsinvalidaddress SendsSIGSEGsignaltouserprocess Userprocess Use p ocesse exits tswith t segmentation seg e tat o fault au t
User Process OS

13;

event

movl

page fault D t t invalid Detect i lid address dd Signal process

Exception Table IA32 (Excerpt)

Exception Number 0 13 14 18 32127 128(0x80) 129255 Description Divideerror Generalprotectionfault Pagefault Machinecheck OSdefined Systemcall OSdefined ExceptionClass Fault Fault Fault Abort Interruptortrap Trap Interruptortrap

Checkpp.183: http://download.intel.com/design/processor/manuals/253665.pdf

Linkers

SomenotesadoptedfromBryantandOHallaron

Example C Program
main.c int buf[2] = {1, 2}; int main() { swap(); return 0; } swap.c extern int buf[]; static int *bufp0 = &buf[0]; static int *bufp1; void swap() { i t temp; int t bufp1 = &buf[1]; temp = *bufp0; *bufp0 = *bufp1; *bufp1 = temp; }

Static Linking
Programsaretranslatedandlinkedusingacompiler driver:
unix>gccO2gopmain.cswap.c unix>./p
main.c Translators (cpp,cc1,as) main o main.o Linker(ld) p Fullylinkedexecutableobjectfile (containscodeanddataforallfunctions definedinmain.candswap.c swap.c Translators (cpp,cc1,as) swap o swap.o Separatelycompiled relocatableobjectfiles Sourcefiles

Why Linkers? Modularity!

Programcanbewrittenasacollectionofsmaller sourcefiles,ratherthanonemonolithicmass. Canbuildlibrariesofcommonfunctions(moreon thislater)
e.g., e g Mathlibrary, library standardClibrary

Why Linkers? Efficiency!

Time:SeparateCompilation
Changeonesourcefile,compile,andthenrelink. Noneedtorecompileothersourcefiles.

Space:Libraries
Commonfunctionscanbeaggregatedintoasinglefile file... Yetexecutablefilesandrunningmemoryimagescontain onlycodeforthefunctionstheyactuallyuse.

What Do Linkers Do?

Step1:Symbolresolution
Programsdefineandreferencesymbols(variablesand functions):
voidswap(){}/*definesymbolswap*/ swap();/*referencesymbolswap*/ int *xp =&x;/*definexp,referencex*/

S Symbol b ld definitions fi iti arestored t d(b (bycompiler) il )i insymbol b l table.

Symboltableisanarrayofstructs Eachentryincludesname,type,size,andlocationofsymbol.

Linkerassociateseachsymbolreferencewithexactly onesymboldefinition.

What Do Linkers Do? (cont.)

Step2:Relocation
Mergesseparatecodeanddatasectionsintosingle sections Relocatessymbolsfromtheirrelativelocationsinthe.o filestotheirfinalabsolutememorylocationsinthe executable. Updatesallreferencestothesesymbolstoreflecttheir newpositions.

Three Kinds of Object Files (Modules)

Relocatable objectfile(.ofile)
Containscodeanddatainaformthatcanbecombinedwith other th relocatable l t bl object bj tfil filesto t form f executable t bl object bj tfil file. Each.ofileisproducedfromexactlyonesource(.c)file

Executableobjectfile
Containscodeanddatainaformthatcanbecopieddirectlyinto yandthenexecuted. memory

Sharedobjectfile(.sofile)
Specialtypeofrelocatable objectfilethatcanbeloadedinto memoryandlinkeddynamically,ateitherloadtimeorruntime. CalledDynamicLinkLibraries(DLLs)byWindows

Packaging Commonly Used Functions`

Howtopackagefunctionscommonlyusedby programmers?
Math,I/O,memorymanagement,stringmanipulation, etc.

Awkward,giventhelinkerframeworksofar:
Option p 1:Putallfunctionsintoasingle g sourcefile
Programmerslinkbigobjectfileintotheirprograms Spaceandtimeinefficient

Option2:Puteachfunctioninaseparatesourcefile
Programmersexplicitlylinkappropriatebinariesintotheir programs Moreefficient,butburdensomeontheprogrammer

Solution: Static Libraries

Staticlibraries(.aarchivefiles)
Concatenaterelatedrelocatableobjectfilesintoasingle filewithanindex(calledanarchive). Enhancelinkersothatittriestoresolveunresolved externalreferencesbylookingforthesymbolsinoneor morearchives. Ifanarchivememberfileresolvesreference,linkinto executable.

Creating Static Libraries

atoi.c Translator atoi.o printf.c Translator printf.o ... random.c Translator random.o

Archiver (ar)

unix> ar rs libc.a libc a \ atoi.o printf.o random.o

libc.a

Cstandardlibrary

Archiver allowsincrementalupdates Recompilefunctionthatchangesandreplace.ofile inarchive.

Commonly Used Libraries

libc.a (theCstandardlibrary)
8MBarchiveof900objectfiles. I/O, I/O memoryallocation, allocation signalhandling, handling stringhandling, handling dataandtime, time random numbers,integermath

libm.a (theCmathlibrary)
1MBarchive hi of f226object bj fil files. floatingpointmath(sin,cos,tan,log,exp,sqrt,)
% ar -t /usr/lib/libc.a | sort fork.o fprintf.o fpu_control.o fputc.o freopen.o fscanf.o fseek o fseek.o fstab.o % ar -t /usr/lib/libm.a | sort e_acos.o e_acosf.o e_acosh.o e_acoshf.o e_acoshl.o e_acosl.o e_asin.o e asinf o e_asinf.o e_asinl.o

Linking with Static Libraries

addvec.o main2.c vector.h Translators (cpp,cc1,as) Relocatable objectfiles main2.o multvec.o

Archiver (ar) libvector.a addvec.o Linker(ld) Fullylinked executableobjectfile libc.a Staticlibraries

printf.o andanyother modulescalledbyprintf.o

p2

Using Static Libraries

Linkersalgorithmforresolvingexternalreferences:
Scan.ofilesand.afilesinthecommandlineorder. Duringthescan,keepalistofthecurrentunresolvedreferences. Aseachnew.oor.afile,obj,isencountered,trytoresolveeach unresolvedreferenceinthelistagainstthesymbolsdefinedin obj. Ifanyentriesintheunresolvedlistatendofscan,thenerror.

Problem:
Commandlineordermatters! Moral:putlibrariesattheendofthecommandline.
unix> gcc -L. libtest.o -lmine unix> gcc -L. L -lmine lmine libtest.o libtest o libtest.o: In function `main': libtest.o(.text+0x4): undefined reference to `libfun'

Shared Libraries
Staticlibrarieshavethefollowingdisadvantages:
Duplicationinthestoredexecutables(everyfunction needstdlibc) Duplicationintherunningexecutables Minorbugfixesofsystemlibrariesrequireeach applicationtoexplicitlyrelink

ModernSolution:SharedLibraries
Object j filesthatcontaincodeanddatathatareloaded andlinkedintoanapplicationdynamically,ateither loadtimeorruntime Alsocalled:dynamiclinklibraries,DLLs,.sofiles

Shared Libraries (cont.)

Dynamiclinkingcanoccurwhenexecutableisfirstloaded andrun(loadtimelinking).
CommoncaseforLinux,handledautomaticallybythedynamic linker(ldlinux.so). StandardClibrary(libc.so) (libc so)usuallydynamicallylinked. linked

Dynamiclinkingcanalsooccurafterprogramhasbegun (runtimelinking). ( g)
InUnix,thisisdonebycallstothedlopen()interface.
Highperformancewebservers. Runtimelibraryinterpositioning

Sharedlibraryroutinescanbesharedbymultiple processes. processes

Moreonthiswhenwelearnaboutvirtualmemory

Dynamic Linking at LoadLoad-time

main2.c vector.h unix> gcc -shared -o libvector.so \ addvec.c multvec.c Translators (cpp,cc1,as) Relocatable object j file f main2.o Linker(ld) Partiallylinked executableobjectfile libc.so libvector.so Relocationandsymbol tableinfo

p2

Loader (execve)

libc.so libvector.so Codeanddata

Fullylinked executable inmemory

Dynamiclinker(ld-linux.so)

Refined View of Memory

Kernelvirtualmemory
0xc0000000

Memory invisibleto usercode %esp (stack pointer)

Userstack (createdatruntime)

Memorymappedregionfor sharedlibraries
0x40000000

brk Runtimeheap (createdby ( ymalloc) Read/writesegment (.data,.bss) Readonlysegment (.init,.text,.rodata)

0x08048000

Loaded from the executable file

Unused

Case Study: Library Interpositioning

Libraryinterpositioning isapowerfullinking techniquethatallowsprogrammerstointercept callstoarbitraryfunctions Interpositioning p g canoccurat:
compiletime
Whenthesourcecodeiscompiled p

linktime
Whentherelocatable objectfilesarelinkedtoforman executableobjectfile

load/runtime
Whenanexecutableobjectfileisloadedintomemory, dynamicallylinked,andthenexecuted.

Some Interpositioning Applications

Security
Confinement(sandboxing)
Interposecallstolibc functions.

Behindthescenesencryption
Automaticallyencryptotherwiseunencryptednetwork connections.

MonitoringandProfiling f
Countnumberofcallstofunctions Characterizecallsitesandargumentstofunctions Malloc tracing
Detectingmemoryleaks Generatingmalloc traces