You are on page 1of 45

Unit OS B: Comparing the Linux

and Windows Kernels

Windows Operating System Internals - by David A. Solomon and Mark E. Russinovich with Andreas Polze
Copyright Notice
2000-2005 David A. Solomon and Mark Russinovich

These materials are part of the Windows Operating


System Internals Curriculum Development Kit,
developed by David A. Solomon and Mark E.
Russinovich with Andreas Polze
Microsoft has licensed these materials from David
Solomon Expert Seminars, Inc. for distribution to
academic organizations solely for use in academic
environments (and not for commercial use)

2
The History of Linux
The real history of Linux starts in 1969, when Ken
Thompson developed the first version of UNIX at Bell
Labs
After Dennis Ritchie, designer of the C programming language,
joined the project it debuted to the research community in an
academic paper in 1974
Bell Labs released the first commercial version in 1976 as UNIX
Version 6 (V6)
UNIX spread throughout universities and in 1978 Bell
Labs released UNIX Time-Sharing System, a version with
portability in mind

3
Linux History Continued
Because Bell Labs distributed UNIX with source code, the
early 1980s saw three major branches grow on the UNIX
tree:
UNIX System III from Bell Labs UNIX Support Group (USG)
UNIX Berkeley Source Distribution (BSD) from the University of
California at Berkeley
Microsofts XENIX
The UNIX market fragmented further in the 1980s,
despite the IEEEs POSIX standard and the X/Open
Groups Portability Guide

4
Linus and Linux
In 1991 Linus Torvalds took a college computer science
course that used the Minix operating system
Minix is a toy UNIX-like OS written by Andrew Tanenbaum as a
learning workbench
Linus wanted to make MINIX more usable, but Tanenbaum
wanted to keep it ultra-simple
Linus went in his own direction and began working on
Linux
In October 1991 he announced Linux v0.02
In March 1994 he released Linux v1.0

5
The History of Windows (NT)
The history of Windows really begins in the mid-1970s,
when Dick Hustvedt, Peter Lipman and David Cutler
designed the VMS operating system for Digitals 32-bit
VAX processor
Digital shipped VMS v1.0 in 1978
Cutler moved to Seattle to open DECWest and worked on
the Digital Mica OS for a new CPU codenamed Prism
12 engineers went with him and the facility grew to 200
In 1988 Digital cancelled the project

6
The History of Windows Continued

Bill Gates wanted a UNIX rival


He hired Cutler and 20 Digital engineers in 1989
The new project was called NT OS/2 because it focused on OS/2
backward compatibility
With the success of Windows 3.0s 1990 release Gates
refocused the project on Windows compatibility
The project renamed to Windows NT
Microsoft released Windows NT 3.1 in August 1993

7
Windows and Linux

Both Linux and Windows are based on


foundations developed in the mid-1970s

1970 1980 1990 2000

1970 1980 1990 2000

8
Comparing the Architectures

Both Linux and Windows are monolithic


All core operating system services run in a shared address space
in kernel-mode
All core operating system services are part of a single module

Linux: vmlinuz
Windows: ntoskrnl.exe

Windowing is handled differently:


Windows has a kernel-mode Windowing subsystem
Linux has a user-mode X-Windowing system

9
Kernel Architectures
Application

Windows

User Mode
Kernel Mode
System Services
Process Management,
Device
Win32 Memory Management,
Drivers
Windowing I/O Management, etc.
Hardware Dependent Code Application

Linux
X-Windows

User Mode
Kernel Mode
System Services
Process Management, Device
Memory Management, Drivers
I/O Management, etc.
Hardware Dependent Code

10
Linux Kernel
Linux is a monolithic but modular system
All kernel subsystems form a single piece of code with no
protection between them
Modularity is supported in two ways:
Compile-time options
Most kernel components can be built as a dynamically
loadable kernel module (DLKM)
DLKMs
Built separately from the main kernel
Loaded into the kernel at runtime and on demand (infrequently
used components take up kernel memory only when needed)
Kernel modules can be upgraded incrementally
Support for minimal kernels that automatically adapt to the
machine and load only those kernel components that are used
11
Windows Kernel

Windows is a monolithic but modular system


No protection among pieces of kernel code and drivers
Support for Modularity is somewhat weak:
Windows Drivers allow for dynamic extension of kernel
functionality
Windows XP Embedded has special tools / packaging rules that
allow coarse-grained configuration of the OS
Windows Drivers are dynamically loadable kernel modules
Significant amount of code run as drivers (including network
stacks such as TCP/IP and many services)
Built independently from the kernel
Can be loaded on-demand
Dependencies among drivers can be specified
12
Comparing Portability

Both Linux and Windows kernels are portable


Mainly written in C
Have been ported to a range of processor architectures
Windows
i486, MIPS, PowerPC, Alpha, IA-64, x86-64
Only x86-64 and IA-64 currently supported
> 64MB memory required
Linux
Alpha, ARM, ARM26, CRIS, H8300, i386, IA-64, M68000,
MIPS, PA-RISC, PowerPC, S/390, SuperH, SPARC, VAX,
v850, x86-64
DLKMs allow for minimal kernels for microcontrollers
> 4MB memory required

13
Comparing Layering, APIs, Complexity

Windows
Kernel exports about 250 system calls (accessed via ntdll.dll)
Layered Windows/POSIX subsystems
Rich Windows API (17 500 functions on top of native APIs)
Linux
Kernel supports about 200 different system calls
Layered BSD, Unix Sys V, POSIX shared system libraries
Compact APIs (1742 functions in Single Unix Specification
Version 3; not including X Window APIs)

14
Comparing Architectures

Processes and scheduling


SMP support
Memory management
I/O
File Caching
Security

15
Process Management
Windows Linux
Process Process is called a Task
Address space, handle
table, statistics and at least Basic Address space,
one thread handle table, statistics
No inherent parent/child Parent/child relationship
relationship
Basic scheduling unit
Threads
Basic scheduling unit Threads
Fibers - cooperative user- No threads per-se
mode threads
Tasks can act like Windows
threads by sharing handle
table, PID and address
space
PThreads cooperative
user-mode threads

16
Scheduling Priorities
Windows
31
Two scheduling classes
Real time (fixed) -
priority 16-31 Fixed
Dynamic - priority 1-15
Higher priorities are
favored 16

Priorities of dynamic 15
threads get boosted on
wakeups I/O
Dynamic
Thread priorities are
never lowered

Windows
17
Scheduling Priorities
Windows Linux
Two scheduling classes Has 3 scheduling classes:
Normal priority 100-139
Real time (fixed) -
priority 16-31 Fixed Round Robin priority
0-99
Dynamic - priority 1-15 Fixed FIFO priority 0-99
Higher priorities are Lower priorities are favored
favored Priorities of normal threads
Priorities of dynamic go up (decay) as they use
threads get boosted on CPU
wakeups Priorities of interactive
threads go down (boost)
Thread priorities are
never lowered

18
Scheduling Priorities (cont)
31 0

Fixed
Fixed FIFO
Fixed Round-Robin
16
15
99
I/O
Dynamic 100
I/O
Normal
CPU
140
0

Windows Linux
19
Linux Scheduling Details
Most threads use a dynamic priority policy
Normal class - similar to the classic UNIX scheduler
A newly created thread starts with a base priority
Threads that block frequently (I/O bound) will have their
priority gradually increased
Threads that always exhaust their time slice (CPU bound) will
have their priority gradually decreased
Nice value sets a threads base priority
Larger values = less priority, lower values = higher priority
Valid nice values are in the range of -20 to +20
Nonprivileged users can only specify positive nice value
Dynamic priority policy threads have static priority zero
Execute only when there are no runnable real-time threads
20
Real-Time Scheduling on Linux
Linux supports two static priority scheduling policies:
Round-robin and FIFO (first in, first out)
Selected with the sched-setscheduler( ) system call
Use static priority values in the range of 1 to 99
Executed strictly in order of decreasing static priority
FIFO policy lets a thread run to completion
Thread needs to indicate completion by calling the sched-yield( )
Round-robin lets threads run for up to one time slice
Then switches to the next thread with the same static priority
RT threads can easily starve lower-prio threads from executing
Root privileges or the CAP-SYS-NICE capability are required for
the selection of a real-time scheduling policy
Long running system calls can cause priority-inversion
Same as in Windows; but cmp. rtLinux
21
Windows Scheduling Details
Most threads run in variable priority levels
Priorities 1-15;
A newly created thread starts with a base priority
Threads that complete I/O operations experience priority
boosts (but never higher than 15)
A threads priority will never be below base priority
The Windows API function SetThreadPriority() sets the
priority value for a specified thread
This value, together with the priority class of the thread's
process, determines the thread's base priority level
Windows will dynamically adjust priorities for non-realtime
threads

22
Real-Time Scheduling on Windows
Windows supports static round-robin scheduling policy
for threads with priorities in real-time range (16-31)
Threads run for up to one quantum
Quantum is reset to full turn on preemption
Priorities never get boosted
RT threads can starve important system services
Such as CSRSS.EXE
SeIncreaseBasePriorityPrivilege required to elevate a threads
priority into real-time range (this privilege is assigned to
members of Administrators group)
System calls and DPC/APC handling can cause priority
inversion

23
Scheduling Timeslices
Windows Linux
The thread timeslice The thread quantum is
(quantum) is 10ms-120ms 10ms-200ms
When quanta can vary, Default is 100ms
has one of 2 values Varies across entire
range based on priority,
Reentrant and
which is based on
preemptible interactivity level
Reentrant and
preemptible
Fixed: 120ms

20ms Background 10ms 200ms

100ms
Foreground: 60ms
24
Virtual Memory Management
Windows Linux
32-bit versions split user- Splits user-mode/kernel-mode
mode/kernel-mode from 2GB/2GB from 1GB/3GB to 3GB/1GB
to 3GB/1GB 2.6 has 4/4 split option where
Demand-paged virtual memory kernel has its own address
space
32 or 64-bits
Demand-paged virtual memory
Copy-on-write
32-bits and/or 64-bits
Shared memory
Copy-on-write
Memory mapped files
Shared memory
0 Memory mapped files
User
0
2GB
User

System
4GB 3GB

System
4GB

25
Physical Memory Management
Windows Linux
Per-process working sets Global working set
Working set tuner adjust management
sets according to memory uses clock algorithm
needs using the clock
algorithm No swapper (the working
No swapper set trimmer code is called
the swap daemon, however)
LRU

Process

Reused Page LRU


LRU
Other Process

Reused Page

26
I/O Management
Windows Linux
Centered around the file object Centered around the vnode
Layered driver architecture No layered I/O model
throughout driver types
Most I/O supports asynchronous
Most I/O is synchronous
operation Only sockets and direct disk
Internal interrupt request level I/O support asynchronous
(IRQL) controls interruptability I/O
Interrupts are split between an Internal interrupt request level
Interrupt Service Routine (ISR) (IRQL) controls interruptability
and a Deferred Procedure Call Interrupts are split between an
(DPC) ISR and soft IRQ or tasklet
Supports plug-and-play
Supports plug-and-play

IRQL
Masked

27
File Caching
Windows Linux
Single global common cache Single global common cache
Virtual file cache Virtual file cache
Caching is at file vs. disk block Caching is at file vs. disk block
level level
Files are memory mapped into Files are memory mapped into
kernel memory kernel memory
Cache allows for zero-copy file Cache allows for zero-copy file
serving serving

File Cache File Cache

File System Driver File System Driver

Disk Driver Disk Driver

28
Security
Windows Linux
Very flexible security model based on Two models:
Access Control Lists
Users are defined with Standard UNIX model
Privileges Access Control Lists (SELinux)
Member groups
Users are defined with:
Security can be applied to any Object
Manager object Capabilities (privileges)
Files, processes, synchronization Member groups
objects,
Supports auditing Security is implemented on an
object-by-object basis
Has no built-in auditing support
Version 2.6 includes Linux Security
Module framework for add-on
security models

29
Monitoring - Linux procfs

Linux supports a number of special filesystems


Like special files, they are of a more dynamic nature and tend to have side
effects when accessed
Prime example is procfs (mounted at /proc)
provides access to and control over various aspects of Linux (I.e.; scheduling
and memory management)
/proc/meminfo contains detailed statistics on the current memory usage of Linux
Content changes as memory usage changes over time

Services for Unix implements procfs on Windows

30
Windows Evolution Towards Linux

Services for Unix 3.5 - really targeted at POSIX, not Linux


POSIX threads, full POSIX subsystem (Interix)
X Window clients+server (X-Win32 LX)
POSIX compatibility in
nfs, NIS, pam
Windows actually
proc-file system for Windows predates Linux and
was one of the original
Configurability / Module Management
design goals
Windows XP Embedded
Target Designer/Component Designer/
Component Management Database
Editions targeting new Application Domains
Windows Compute Cluster Server 2003

31
Linuxs Evolution Towards Windows

I/O processing
Kernel reentrancy
Kernel preemptibility
Per-processor memory allocation
O(1) scheduler and per-CPU ready queues
Zero-Copy SendFile
Wake-One socket semantics
Asynchronous I/O
Light-weight synchronization

32
I/O Processing
Linux 2.2 had the notion of bottom halves (BH) for low-
priority interrupt processing
Fixed number of BHs
Only one BH of a given type could be active on a SMP
Linux 2.4 introduced tasklets, which are non-preemptible
procedures called with interrupts enabled
Tasklets are the equivalent of Windows Deferred
Procedure Calls (DPCs)

33
Kernel Reentrancy

Mark Russinovichs April 1999 Windows NT Magazine article, Linux


and the Enterprise, pointed out that much of the Linux 2.2 was not
reentrant
cpu 1
Non-reentrant
cpu 2

Reentrant cpu 1
cpu 2

Ingo Molnar stated in rebuttal: Time Saved

his example is a clear red herring.


A month later he made all major paths reentrant

34
Kernel Preemptibility
A preemptible kernel is more responsive to high-priority
tasks
Through the base release of v2.4 Linux was only
cooperatively preemptible
There are well-defined safe places where a thread running in the
kernel can be preempted
The kernel is preemptible in v2.4 patches and v2.6
Windows NT has always been preemptible

35
Scheduling
The Linux 2.4 scheduler is O(n)
If there are 10 active tasks, it scans 10 of them in a list in order to
decide which should execute next
This means long scans and long durations under the scheduler lock

Ready
List 103 112 112 101

Highest
Priority
Task

36
Scheduling

Linux 2.6 has a revamped scheduler thats O(1) from Ingo Molnar
that:
Calculates a tasks priority at the time it makes scheduling decision
Has per-CPU ready queues where the tasks are pre-sorted by priority

Highest-priority 101
Non-empty Queue

103

112 112

37
Scheduling

Windows NT has always had an O(1) scheduler based


on pre-sorted thread priority queues
Server 2003 introduced per-CPU ready queues
Linux load balances queues
Windows does not
Not seen as an issue in performance testing by Microsoft
Applications where it might be an issue are expected to use affinity

38
Zero-Copy Sendfile
Linux 2.2 introduced Sendfile to efficiently send file data over a socket
I pointed out that the initial implementation incurred a copy operation,
even if the file data was cached
Linux 2.4 introduced zero-copy Sendfile
Windows NT pioneered zero-copy file sending with TransmitFile, the
Sendfile equivalent, in Windows NT 4

File Data File Data


1-Copy Buffer
0-Copy
Buffer

Network Network Network


Network Network
Adapter
Driver Driver
Buffer
39
Wake-one Socket Semantics

Linux 2.2 kernel had the thundering herd or


overscheduling problem
In a network server application there are typically several
threads waiting for a new connection
In v2.2 when a new connection came in all the waiters would
race to get it
Ingo Molnars response:
5/2/99: here he again forgets to _prove_ that overscheduling
happens in Linux.
5/7/99: as of 2.3.1 my wake-one implementation and
waitqueues rewrite went in
In Linux 2.4 only one thread wakes up to claim the new
connection
Windows NT has always had wake-1 semantics
40
Asynchronous I/O
Linux 2.2 only supported asynchronous I/O on socket
connect operations and ttys
Linux 2.6 adds asynchronous I/O for direct-disk access
AIO model includes efficient management of asynchronous I/O
Also added alternate epoll model
Useful for database servers managing their database on a
dedicated raw partition
Database servers that manage a file-based database suffer from
synchronous I/O
Windows I/O is inherently asynchronous
Windows has had completion ports since NT 3.5
More advanced form of AIO

41
Light-Weight Synchronization

Linux 2.6 introduces Futexes


Theres only a transition to kernel-mode when theres
contention
Windows has always had CriticalSections
Same behavior
Futexes go further:
Allow for prioritization of waits
Works interprocess as well

42
A Look at the Future
The kernel architectures are fundamentally similar
There are differences in the details
Linux implementation is adopting more of the good ideas used in
Windows
For the next 2-4 years Windows has and will maintain an
edge
Linux is still behind on the cutting edge of performance tricks
Large performance team and lab at Microsoft has direct ties into
the kernel developers
As time goes on the technological gap will narrow
Open Source Development Labs (OSDL) will feed performance
test results to the kernel team
IBM and other vendors have Linux technology centers
Squeezing performance out of the OS gets much harder as the OS
gets more tuned

43
Linux Technology Unknowns

Linux kernel forking


RedHat has already done it: Red Hat Enterprise Server v3.0 is
Linux 2.4 with some Linux 2.6 features
Backward compatibility philosophy
Linus Torvalds makes decisions on kernel APIs and
architecture based on technical reasons, not business reasons

44
Further Reading
Transaction Processing Council: www.tpc.org
SPEC: www.spec.org
NT vs Linux benchmarks: www.kegel.com/nt-linux-
benchmarks.html
The C10K problem: http://www.kegel.com/c10k.html
Linus Torvalds home: http://www.osdl.org/
Linux Kernel Archives: http://www.kernel.org/
Linux history: http://www.firstmonday.dk/issues/issue5_11/moon/
Veritest Netbench result:
http://www.veritest.com/clients/reports/microsoft/ms_netbench.pdf
Mark Russinovichs 1999 article, Linux and the Enterprise:
http://www.winntmag.com/Articles/Index.cfm?ArticleID=5048
The Open Group's Single UNIX Specification:
http://www.unix.org/version3/
45

You might also like