Set 1: Introduction

CSCE 668
DISTRIBUTED ALGORITHMS AND
SYSTEMS

CSCE 668

Spring 2014
Prof. Jennifer Welch

Distributed Systems
2

Distributed systems have become ubiquitous:

share resources
communicate
increase performance

speed
fault tolerance

Characterized by

independent activities (concurrency)

loosely coupled parallelism (heterogeneity)
inherent uncertainty
Set 1: Introduction CSCE 668

Uncertainty in Distributed
Systems

Uncertainty comes from

differing processor speeds

varying communication delays
(partial) failures
multiple input streams and interactive
behavior

Set 1: Introduction CSCE 668

Reasoning about Distributed

Systems

Uncertainty makes it hard to be confident

that system is correct
To address this difficulty:

identify and abstract fundamental problems

state problems precisely
design algorithms to solve problems
prove correctness of algorithms
analyze complexity of algorithms (e.g., time,
space, messages)
prove impossibility results and lower bounds
Set 1: Introduction CSCE 668

Potential Payoff of Theoretical

Paradigm

careful specifications clarify intent

increased confidence in correctness
if abstracted well then results are
relevant in multiple situations
indicate inherent limitations

cf. NP-completeness

Set 1: Introduction CSCE 668

Application Areas
6

These areas have provided classic problems

in distributed/concurrent computing:

operating systems
(distributed) database systems
software fault-tolerance
communication networks
multiprocessor architectures

Newer application areas:

cloud computing
mobile computing,
Set 1: Introduction CSCE 668

Course Overview: Part I

(Fundamentals)

Introduce two basic communication

models:

message passing
shared memory

and two basic timing models:

synchronous
asynchronous

Set 1: Introduction CSCE 668

Course Overview: Basic

Models
Message passing
synchronous Yes

asynchronous

Yes

Shared memory

No

Yes

(Synchronous shared memory model is PRAM)

Set 1: Introduction CSCE 668

Course Overview: Part I

9

Covers the canonical problems and

issues:

graph algorithms (Ch 2)

leader election (Ch 3)
mutual exclusion (Ch 4)
fault-tolerant consensus (Ch 5)
causality and time (Ch 6)

Set 1: Introduction CSCE 668

10

Course Overview: Part II

(Simulations)

Here "simulations" means abstractions,

or techniques for making it easier to
program, by making one model appear
to be an easier model. For example:

broadcast and multicast (Ch 8)

distributed shared memory (Ch 9)
stronger kinds of shared variables (Ch 10)
more synchrony (Chs 11, 13)
more benign faults (Ch 12)

Set 1: Introduction CSCE 668

Course Overview: Part II

11

For each of the techniques:

describe algorithms for implementing it

analyze the cost of these algorithms
explore limitations

mention applications that use the

techniques

Set 1: Introduction CSCE 668

12

Course Overview: Part III

(Advanced Topics)

Push further in some directions already

introduced:

randomized algorithms (Ch 14)

stronger kinds of shared objects of arbitrary
type (Ch 15)
what kinds of problems are solvable in
asynchronous systems (Ch 16)
failure detectors (Ch 17)
self-stabilization

Set 1: Introduction CSCE 668

13

Relationship of Theory to
Practice

time-shared operating systems: issues

relating to (virtual) concurrency of
processes such as

also arise in distributed systems

MIMD multiprocessors:

mutual exclusion
deadlock

no common clock => asynchronous model

common clock => synchronous model

loosely coupled networks, such as Internet,

=> asynchronous model
Set 1: Introduction CSCE 668

14

Relationship of Theory to
Practice

Failure models:

crash: faulty processor just stops.

Idealization of reality.
Byzantine (arbitrary): conservative
assumption, fits when failure model is
unknown or malicious
self-stabilization: algorithm automatically
recovers from transient corruption of state;
appropriate for long-running applications
Set 1: Introduction CSCE 668

Message-Passing Model
15

processors are p0, p1, , pn-1 (nodes of

graph)
bidirectional point-to-point channels
(undirected edges of graph)
each processor labels its incident
channels 1, 2, 3,; might not know who
is at other end

Set 1: Introduction CSCE 668

Message-Passing Model
16

p3

1 p
0
3
2
2
p2
1

2
p1

Set 1: Introduction CSCE 668

17

Modeling Processors and

Channels

Processor is a state machine including

Channel directed from processor pi to

processor pj is modeled in two pieces:

local state of the processor

mechanisms for modeling channels

outbuf variable of pi and

inbuf variable of pj

Outbuf corresponds to physical channel,

inbuf to incoming message queue
Set 1: Introduction CSCE 668

18

Modeling Processors and

Channels
inbuf[1]

outbuf[2]

p1's local
variables

p2's local
variables
outbuf[1]

inbuf[2]

Pink area (local vars + inbuf) is accessible state

for a processor.
Set 1: Introduction CSCE 668

Configuration
19

Vector of processor states (including

outbufs, i.e., channels), one per
processor, is a configuration of the
system
Captures current snapshot of entire
system: accessible processor states
(local vars + incoming msg queues) as
well as communication channels.
Set 1: Introduction CSCE 668

Deliver Event
20

Moves a message from sender's

outbuf to receiver's inbuf; message
will be available next time receiver
takes a step
p1

p1

m3 m2 m1

m3 m2

p2

m1

p2

Set 1: Introduction CSCE 668

Computation Event
21

Occurs at one processor

Start with old accessible state (local vars
+ incoming messages)
Apply transition function of processor's
state machine; handles all incoming
messages
End with new accessible state with empty
inbufs, and new outgoing messages

Set 1: Introduction CSCE 668

Computation Event
22

b
a
c

old
local
state

d e

new
local
state

pink indicates accessible state: local vars and incoming m

white indicates outgoing msg buffers
Set 1: Introduction CSCE 668

Execution
23

Format is
config, event, config, event, config,
in first config: each processor is in initial
state and all inbufs are empty
for each consecutive (config, event, config),
new config is same as old config except:

if delivery event: specified msg is transferred

from sender's outbuf to receiver's inbuf
if computation event: specified processor's state
(including outbufs) change according to
transition function
Set 1: Introduction CSCE 668

Admissibility
24

Definition of execution gives some basic

"syntactic" conditions.

Sometimes we want to impose additional

constraints

usually safety conditions (true in every finite prefix)

usually liveness conditions (eventually something

happens)

Executions satisfying the additional constraints

are admissible. These are the executions that
must solve the problem of interest.

Definition of admissible can change from context to

context, depending on details of what we are
modeling
Set 1: Introduction CSCE 668

Asynchronous Executions
25

An execution is admissible for the

asynchronous model if

every message in an outbuf is eventually

delivered
every processor takes an infinite number of steps

No constraints on when these events take

place: arbitrary message delays and
relative processor speeds are not ruled out
Models reliable system (no message is lost
and no processor stops working)
Set 1: Introduction CSCE 668

Example: Flooding
26

Describe a simple flooding algorithm as a

collection of interacting state machines.
Each processor's local state consists of
variable color, either red or green
Initially:

p0: color = green, all outbufs contain M

others: color = red, all outbufs empty

Transition: If M is in an inbuf and color =

red, then change color to green and send
M on all outbufs
Set 1: Introduction CSCE 668

Example: Flooding
27

p0

p2

M
deliver event
at p1 from p0

p1

p0

p0

p2

p1

p0

M
p2

p1
M

computation
event by p1

M
deliver event
at p2 from p1

p2

p1
M

computation
event by p2

Set 1: Introduction CSCE 668

Example: Flooding (cont'd)

28

M
M

p0

M
M

p2

deliver event
at p1 from p2

p1
M

M
M

M
p2

M
p1

deliver event
at p0 from p1

p1
M

p0
M

p2

p0

p2

p0

computation
event by p1

M
p1

etc. to
deliver
rest of
msgs

Set 1: Introduction CSCE 668

Nondeterminism
29

The previous execution is not the only

admissible execution of the Flooding
algorithm on that triangle.
There are several, depending on the
order in which messages are delivered.
For instance, the message from p0 could
arrive at p2 before the message from p1
does.
Set 1: Introduction CSCE 668

Termination
30

For technical reasons, admissible

executions are defined as infinite.
But often algorithms terminate.
To model algorithm termination, identify
terminated states of processors: states
which, once entered, are never left
Execution has terminated when all
processors are terminated and no messages
are in transit (in inbufs or outbufs)
Set 1: Introduction CSCE 668

31

Termination of Flooding
Algorithm

Define terminated processor states as

those in which color = green.

Set 1: Introduction CSCE 668

32

Message Complexity
Measure

Message complexity: maximum

number of messages sent in any
admissible execution

This is a worst-case measure.

Later we will mention average-case
measures.

Set 1: Introduction CSCE 668

33

Message Complexity of
Flooding Algorithm

Message complexity: one message is

sent over each edge in each direction.
So number is 2m, where m = number
of edges.

Set 1: Introduction CSCE 668

Time Complexity Measure

34

How can we measure time in asynchronous

executions?
Produce a timed execution from an execution
by assigning non-decreasing real times to
events such that time between sending and
receiving any message is at most 1.
Essentially normalizes the greatest message
delay in an execution to be one time unit; still
allows arbitrary interleavings of events.
Time complexity: maximum time until
termination in any timed admissible execution.
Set 1: Introduction CSCE 668

35

Time Complexity of
Flooding Algorithm

Recall that terminated processor states

are those in which color = green.

Time complexity: diameter + 1 time

units. (A node turns green once a
"chain" of messages has reached it
from p0.)

Diameter of a graph is the maximum, over

all nodes v and w in the graph of the
shortest path from v to w
Set 1: Introduction CSCE 668

36

Synchronous Message Passing

Systems

An execution is admissible for the

synchronous model if it is an infinite
sequence of "rounds"
What is a "round"?
It is a sequence of deliver events that
move all messages in transit into
inbuf's, followed by a sequence of
computation events, one for each
processor.
Set 1: Introduction CSCE 668

37

Synchronous Message Passing

Systems

The new definition of admissible

captures lockstep unison feature of
synchronous model.
This definition also implies

every message sent is delivered

every processor takes an infinite number of
steps.

Time is measured as number of rounds

until termination.
Set 1: Introduction CSCE 668

38

Example of Synchronous
Model

Suppose flooding algorithm is executed in

synchronous model on the triangle.
Round 1:

deliver M to p1 from p0
deliver M to p2 from p0
p0 does nothing (as it has no incoming messages)
p1 receives M, turns green and sends M to p0 and
p1
p2 receives M, turns green and sends M to p0 and
p1
Set 1: Introduction CSCE 668

39

Example of Synchronous
Model

Round 2:

deliver M to p0 from p1
deliver M to p0 from p2
deliver M to p1 from p2
deliver M to p2 from p1
p0 does nothing since its color variable is already
green
p1 does nothing since its color variable is already
green
p2 does nothing since its color variable is already
green
Set 1: Introduction CSCE 668

40

Example of Synchronous
Model
p
M

p0

p2

M
p1

M
round 1
events

p2

M M

p1

round 2
events

p0
p2

p1
Set 1: Introduction CSCE 668

41

Complexity of Synchronous
Flooding Algorithm

Just consider executions that are

admissible w.r.t. synchronous model (i.e.,
that satisfy the definition of synchronous
model)
Time complexity is diameter + 1
Message complexity is 2m
Same as for asynchronous case.
Not true for all algorithms though
Set 1: Introduction CSCE 668