Chapter 2 - Communication

Introduction
interprocess communication is at the heart of all distributed systems communication in distributed systems is based on message passing as offered by the underlying network as opposed to using shared memory modern distributed systems consist of thousands of processes scattered across an unreliable network such as the Internet unless the primitive communication facilities of the network are replaced by more advanced ones, development of large scale Distributed Systems becomes extremely difficult

2.1 Layered Protocols

two computers, possibly from different manufacturers, must be able to talk to each other for such a communication, there has to be a standard The ISO OSI (Open Systems Interconnection) Reference Model is one of such standards - 7 layers TCP/IP protocol suite is the other; has 4 or 5 layers OSI Open to connect open systems or systems that are open for communication with other open systems using standard rules that govern the format, contents, and meaning of the messages sent and received these rules are called protocols two types of transport layer protocols: connection-oriented and connectionless a conversation occurs between a sender and a receiver at each layer 3

 Middleware Protocols a middleware is an application that contains general-purpose protocols to provide services example of middleware services authentication and authorization services distributed transactions (commit protocols; locking mechanisms) middleware communication protocols (calling a procedure or invoking an object remotely, synchronizing streams for real-time data, multicast services) - see later in this Chapter hence an adapted reference model for networked communications is required

an adapted reference model for networked communication

2.2 Remote Procedure Call

the first distributed systems were based on explicit message exchange between processes through the use of explicit send and receive procedures; but do not allow access transparency in 1984, Birrel and Nelson introduced a different way of handling communication: RPC it allows a program to call a procedure located on another machine simple and elegant, but there are implementation problems the calling and called procedures run in different address spaces parameters and results have to be exchanged; what if the machines are not identical? what happens if both machines crash?

 Client and Server Stubs RPC would like to make a remote procedure call look the same as a local one; it should be transparent, i.e., the calling procedure should not know that the called procedure is executing on a different machine or vice versa

principle of RPC between a client and server program

when a program is compiled, it uses different versions of library functions called client stubs a server stub is the server-side equivalent of a client stub

 Steps of a Remote Procedure Call 1. Client procedure calls client stub in the normal way 2. Client stub builds a message and calls the local OS (packing parameters into a message is called parameter marshaling) 3. Client's OS sends the message to the remote OS 4. Remote OS gives the message to the server stub 5. Server stub unpacks the parameters and calls the server 6. Server does the work and returns the result to the stub 7. Server stub packs it in a message and calls the local OS 8. Server's OS sends the message to the client's OS 9. Client's OS gives the message to the client stub 10. Stub unpacks the result and returns to client hence, for the client remote services are accessed by making ordinary (local) procedure calls; not by calling send and receive
server machine vs server process; client machine vs client process
8

 Parameter Passing 1. Passing Value Parameters e.g., consider a remote procedure add(i, j), where i and j are integer parameters

steps involved in doing remote computation through RPC

 the above discussion applies if the server and the client machines are identical but that is not the case in large distributed systems, there can be differences in: data representation (e.g., IBM mainframes use EBCDIC whereas IBM PCs use ASCII) representing integers(1s complement or 2s complement) and floating-point numbers byte numbering (from right to left in Pentium called little endian and left to right in SPARC, big endian) e.g.

consider a procedure with two parameters, an integer and a four-character string; each one 32-bit word (5, JILL)
the sender is Intel and the receiver is SPARC
10

original message on the Pentium (the numbers in boxes indicate the address of each byte)

the message after receipt on the SPARC; wrong integer (224+226 = 83886080), but correct string

11

 one approach is to invert the bytes of each word after receipt

the message after being inverted (correct integer but wrong string)

additional information is required to tell which is an integer and which is a string

12

2. Passing Reference Parameters assume the parameter is a pointer to an array copy the array into the message and send it to the server the server stub can then call the server with a pointer to this array the server then makes any changes to the array and sends it back to the client stub which copies it to the client this is in effect call-by-copy/restore optimization of the method one of the copy operations can be eliminated if the stub knows whether the parameter is input or output to the server if it is an input to the server (e.g., in a call to write), it need not be copied back if it is an output, it need not be sent over in the first place; only send the size the above procedure can handle pointers to simple arrays and structures, but difficult to generalize it to an arbitrary data structure

13

 Parameter Specification and Stub Generation the caller and the callee need to use the same protocol (format of messages) and the same steps; with such rules the client and server stubs can assemble, communicate, and interpret messages correctly other issues that need the agreement of the client and the server how are simple data structures like integers (e.g. 2s complement), characters (e.g. 16-bit Unicode), Booleans, ... represented? endianess which transport protocol to use - the connection-oriented TCP or the unreliable connectionless UDP

14

 Extended RPC Models to solve some of the shortcomings of the original model no need of network communication if server and client are on the same machine no need of blocking for the client in some cases a. Doors the original RPC model assumes that the caller and the callee can communicate only by means of passing messages over a network; what if they are colocated on the same machine? a door is a generic name for a procedure in the address space of a server process that can be called by a process colocated with the server support from the local OS is required

15

1. the server process registers a door before it can be called (door_create) and a name is attached to it 2. a client calls a door by a system call (door_call) including all parameters 3. results are returned by the system call door_return

3 1 2

the principle of using doors as IPC mechanism

16

 benefit: they allow the use of a single mechanism (procedure calls) for communication disadv: application developers have to be aware of where a procedure is located; is it local within the current process local to a different process on the same machine a remote process

17

b. Asynchronous RPC if there is no need to block the client until it gets a reply two cases 1. if there is no result to be returned e.g., adding entries in a database, ... the server immediately sends an ack promising that it will carryout the request the client can now proceed without blocking

a) the interconnection between client and server in a traditional RPC b) the interaction using asynchronous RPC 18

2. if the result can be collected later e.g., prefetching network addresses of a set of hosts, ... the server immediately sends an ack promising that it will carryout the request the client can now proceed without blocking the server later sends the result

a client and server interacting through two asynchronous RPCs

19

 the above method combines two asynchronous RPCs and is sometimes called deferred synchronous RPC variants of asynchronous RPC let the client continue without waiting even for an ack, called one-way RPC problem: if reliability of communication is not guaranteed

20

 DCE (Distributed Computing Environment) RPC a middleware and an example RPC system developed by OSF (Open Software Foundation), now The Open Group it is designed to execute as a layer of abstraction between existing OSs and distributed applications the Open Group sells the source code and vendors integrate it into their systems it uses the client-server programming model and communication is by means of RPCs services distributed file service: a worldwide file system that provides a transparent way of accessing files directory service: to keep track of the location of all resources in the system (machines, printers, data, servers, ...); a process can ask for a resource without knowing its location security service: for protecting resources; access is only through authorization 21

 distributed time service: to maintain clocks on different machines synchronized

Steps in writing a Client and a Server in DCE RPC the system consists of languages, libraries, daemons, utility programs, ... for writing clients and servers IDL (Interface Definition Language) is the interface language - the glue that holds everything together it contains type definitions, constant declarations and what the procedures do (only their syntax)

22

Edit file

Uuidgen generates a prototype IDL file with a globally unique interface identifier the IDL file is edited (filling the names of procedures and parameters) and the IDL compiler is called to generate 3 files the application writer writes the client and server codes and are then 23 compiled and linked together with the stubs

 Binding a Client to a Server in DCE RPC for a client to call a server, the server must be registered (1 & 2) the registration allows the client to locate the server and bind to it the DCE daemon maintains a table (server, endpoint) and the protocols the server uses the directory server maintains the locations of all resources in the system (machines, servers, data,, ...)

24

 two steps for server location locate the servers machine (3) locate the server process on that machine (which has what is called an endpoint or port) (4)

25

2.3 Remote Object (Method) Invocation (RMI) resulted from object-based technology that has proven its value in developing nondistributed applications it is an expansion of the RPC mechanisms it enhances distribution transparency as a consequence of an object that hides its internal from the outside world by means of a well-defined interface Distributed Objects an object encapsulates data, called the state, and the operations on those data, called methods methods are made available through interfaces the state of an object can be manipulated only by invoking methods this allows an interface to be placed on one machine while the object itself resides on another machine; such an organization is referred to as a distributed object the state of an object is not distributed, only the interfaces are; such objects are also referred to as remote objects 26

 the implementation of an objects interface is called a proxy (analogous to a client stub in RPC systems) it is loaded into the clients address space when a client binds to a distributed object tasks: a proxy marshals method invocation into messages and unmarshals reply messages to return the result of the method invocation to the client a server stub, called a skeleton, unmarshals messages and marshals replies

27

common organization of a remote object with client-side proxy

28

 Binding a Client to an Object a process must first bind to an object before invoking its methods, which results in a proxy being placed in the processs address space binding can be implicit (directly invoke methods using only a reference to an object) or explicit (by calling a special function) an object reference could contain network address of the machine where the object resides endpoint of the server an identification of which object the protocol used ...

29

Distr_object* obj_ref; obj_ref = ; obj_refdo_something(); Distr_object obj_ref; Local_object* obj_ptr; obj_ref = ; obj_ptr = bind(obj_ref); obj_ptrdo_something();

// Declare a systemwide object reference // Initialize the reference to a distributed object // Implicitly bind and invoke a method (a) // Declare a systemwide object reference // Declare a pointer to local objects // Initialize the reference to a distributed object // Explicitly bind and obtain a pointer to the local proxy // Invoke a method on the local proxy (b)

a) an example with implicit binding using only global references b) an example with explicit binding using global and local references

30

 Parameter Passing there are two situations when invoking a method with object reference as a parameter: the object can be local or remote to the client local object: a copy of the object is passed; this means the object is passed by value remote object: copy and pass the reference of the object as a value parameter; this means the object is passed by reference

31

the situation when passing an object by reference or by value

two examples: DCE Remote Objects Java RMI

32

2.4 Message Oriented Communication RPCs and RMIs are not adequate for all distributed system applications the provision of access transparency may be good but they have semantics that is not adequate for all applications example problems they assume that the receiving side is not running at the time of communication a client is blocked until its request has been processed

33

 communication can be persistent or transient asynchronous or synchronous persistent: a message that has been submitted for transmission is stored by the communication system as long as it takes to deliver it to the receiver e.g., email delivery, snail mail delivery

persistent communication of letters back in the days of the Pony Express

34

 transient: a message that has been submitted for transmission is stored by the communication system only as long as the sending and receiving applications are executing asynchronous: a sender continues immediately after it has submitted its message for transmission synchronous: the sender is blocked until its message is stored in a local buffer at the receiving host or delivered to the receiver the different types of communication can be combined persistent asynchronous: e.g., email transient asynchronous: e.g., UDP, asynchronous RPC in general there are six possibilities
Persistent Asynchronous Synchronous Transient message-oriented; three forms
35

persistent asynchronous communication

persistent synchronous communication

36

transient asynchronous communication

receipt-based transient synchronous communication weakest form; the sender is blocked until the message is stored in a local buffer at the receiving host
37

delivery-based transient synchronous communication at message delivery the sender is blocked until the message is delivered to the receiver for further processing

response-based transient synchronous communication strongest form; the sender is blocked until it receives a reply message from the receiver

38

 Message-Oriented Transient Communication many applications are built on top of the simple messageoriented model offered by the transport layer standardizing the interface of the transport layer by providing a set of primitives allows programmers to use messaging protocols they also allow porting applications Examples of message-oriented transient communication systems: Berkley sockets and the Message-Passing Interface (MPI)

39

 Message-Oriented Persistent Communication there are message-oriented middleware services, called message-queuing systems or Message-Oriented Middleware (MOM) they support persistent asynchronous communication they have intermediate-term storage capacity for messages, without requiring the sender or the receiver to be active during message transmission message transfer may take minutes instead of seconds or milliseconds Message-oriented persistent communication depends on the message queuing model in which applications communicate by inserting messages in specific queues Example: IBM MQSeries

40

2.5 Stream Oriented Communication

until now, we focused on exchanging independent and complete units of information time has no effect on correctness; a system can be slow or fast however, there are communications where time has a critical role Multimedia media storage, transmission, interchange, presentation, representation and perception of different data types: text, graphics, images, voice, audio, video, animation, ... movie: video + audio + multimedia: handling of a variety of representation media

41

 The Challenge new applications multimedia will be pervasive in few years (as graphics) storage and transmission e.g., 2 hours uncompressed HDTV (19201080) movie: 1.12 TB (19201080x3x25x60x60x2) videos are extremely large, even after compressed continuous delivery e.g., 30 frames/s (NTSC), 25 frames/s (PAL) for video guaranteed Quality of Service admission control search can we look at 100 videos to find the proper one?
42

 Types of Media
two types discrete media: text, executable code, graphics, images; temporal relationships between data items are not fundamental to correctly interpret the data continuous media: video, audio, animation; temporal relationships between data items are fundamental to correctly interpret the data a data stream is a sequence of data units and can be applied to discrete as well as continuous media stream-oriented communication provides facilities for the exchange of time-dependent information (continuous media) such as audio and video streams

43

 timing in transmission modes asynchronous transmission mode: data items are transmitted one after the other, but no timing constraints; e.g. text transfer synchronous transmission mode: a maximum end-to-end delay defined for each data unit; it is possible that data can be transmitted faster than the maximum delay, but not slower isochronous transmission mode: maximum and minimum end-to-end delay are defined; also called bounded delay jitter; applicable for distributed multimedia systems a continuous data stream can be simple or complex simple stream: consists of a single sequence of data; e.g., mono audio, video only (only visual frames) complex stream: consists of several related simple streams that must be synchronized; e.g., stereo audio, video consisting of audio and video (may also contain subtitles, translation to other languages, ...)

44

movie as a set of simple streams

45

 a stream can be considered as a virtual connection between a source and a sink the source or the sink could be a process or a device

setting up a stream between two processes across a network

setting up a stream directly between two devices

46

 the data stream can also be multicasted to several receivers if devices and the underlying networks have different capabilities, the stream may be filtered, generally called adaptation

an example of multicasting a stream to several receivers

47

 Quality of Service (QoS) QoS requirements describe what is needed from the underlying distributed system and network to ensure acceptable delivery; e.g. viewing experience of a user for continuous data, the concerns are timeliness: data must be delivered in time volume: the required throughput must be met reliability: a given level of loss of data must not be exceeded quality of perception; highly subjective

48

 QoS Dimensions timeliness dimensions latency (maximum delay between consecutive frames) start-up latency (maximum delay before starting a presentation) jitter (delay variance) volume dimensions throughput in frames/sec or bits/sec or bytes/sec reliability dimensions MTBF (Mean Time Between Failure) of disks MTTR (Mean Time To Repair) error rates on the telecommunication lines

49

 QoS Requirements deterministic precise values or ranges e.g., latency must be between 45 and 55 ms probabilistic probability of the required QoS e.g., latency should be < 50 ms for 95% of the frames stochastic distributions e.g., frame arrival should follow normal distribution with mean interval-time of 40 ms and 5 ms variance classes e.g., guaranteed and best effort

50

 QoS Management can be static or dynamic Static QoS Management Functions specification e.g., deterministic range for timeliness, volume and reliability categories negotiation the application may accept lower level of QoS for lower cost admission control if this test is passed, the system has to guarantee the promised QoS resource reservation may be necessary to provide guaranteed QoS

51

 Dynamic QoS Management Functions monitoring notices deviation from QoS level at a certain level of granularity (e.g., every 100 ms) policing detect participants not keeping themselves to the contract e.g., source sends faster than negotiated (e.g., 25 fps) maintenance sustaining the negotiated QoS e.g., the system requires more resources renegotiation client tries to adapt may be can accept lower QoS

52

 QoS requirements can be specified using flow specification containing bandwidth requirements, transmission rates, delays, ... e.g. by Partridge (1992) it uses the token bucket algorithm which specifies how the stream will shape its network traffic (in fact the leaky bucket, as used in networking) the idea is to shape bursty traffic into fixed-rate traffic by averaging the data rate packets may be dropped if the bucket is full the input rate may vary, but the output rate remains constant

53

the principle of a token bucket algorithm

54

 Specifying QoS
Characteristics of the Input Maximum data unit size (bytes) Token bucket rate (bytes/sec) Token bucket size (bytes) Maximum transmission rate (bytes/sec) Service Required Loss sensitivity (bytes) Loss interval (sec) the above two specify the maximum acceptable loss rate, e.g. 1 byte per minute Burst loss sensitivity (data units); how many consecutive data units may be lost Minimum delay noticed (sec); how long the network can delay the delivery Maximum delay variation (sec); maximum tolerated jitter Quality of guarantee; continue or dont establish a stream if the communication system can not provide the required service a flow specification 55

 problem in flow specification an application may not know its requirements how can a user (human) specify quality using the various parameters? usually very difficult may be provide defaults for various streams as high, medium, low quality Setting up a Stream resources such as bandwidth, buffers, processing power must be reserved once a flow specification is made one such protocol is RSVP - Resource reSerVation Protocol it is a transport-level protocol for enabling resource reservation in network routers

56

the basic organization of RSVP for resource reservation in a distributed system

57

 Stream Synchronization how to maintain temporal relations between streams, e.g., lip synchronization two approaches 1. explicitly by operating on the data units of simple streams; the responsibility of the application

the principle of explicit synchronization on the level of data units

58

2. through a multimedia middleware that offers a collection of interfaces for controlling audio and video streams as well as devices such as monitors, cameras, microphones, ...

the principle of synchronization as supported by high-level interfaces

59