Distributed Systems

RPC and Client-Server Computing

Remote Procedure Call

Basic concepts Implementation issues, usual optimizations Where are the costs? Reliability and consistency Multithreading debate

A brief history of RPC

Introduced by Birrell and Nelson in 1985 Pre-RPC: Most applications were built directly over the Internet primitives Their idea: mask distributed computing system using a transparent abstraction
Looks like normal procedure call Hides all aspects of distributed interaction Supports an easy programming model

Today, RPC is the core of many distributed systems

More history
Early focus was on RPC environments Culminated in DCE (Distributed Computing Environment), standardizes many aspects of RPC Then emphasis shifted to performance, many systems improved by a factor of 10 to 20 Today, RPC often used in object-oriented systems employing CORBA or COM standards.
and now SOAP & web services

Reliability & correctness issues are more evident than in the past.

The basic RPC protocol

client binds to server server registers with name service

The basic RPC protocol

client binds to server prepares, sends request receives request server registers with name service

The basic RPC protocol

client binds to server prepares, sends request receives request invokes handler server registers with name service

The basic RPC protocol

client binds to server prepares, sends request receives request invokes handler sends reply server registers with name service

The basic RPC protocol

client binds to server prepares, sends request receives request invokes handler sends reply unpacks reply packing/unpacking done by RPC software --Programmer doesnt know details, only makes calls to functions
--All

server registers with name service

Compilation stage
Server defines and exports a header file giving interfaces it supports and arguments expected. Uses interface definition language (IDL) Client includes this information Client invokes server procedures through stubs
provides interface identical to the server version responsible for building the messages and interpreting the reply messages (i.e., for packing and unpacking) passes arguments by value, never by reference may limit total size of arguments, in bytes

Binding stage
Occurs when client and server program first start execution Server registers its network address with name directory, perhaps with other information Client scans directory to find appropriate server Depending on how RPC protocol is implemented, may make a connection to the server, but this is not mandatory

Data in messages
We say that data is marshalled into a message and demarshalled from it Representation needs to deal with byte ordering issues (big-endian versus little endian), strings (some CPUs require padding), alignment, etc Must encode info in the messages that allows other side to understand the message
E.g, how to find start of an object in the message Both sides must agree in advance on packed representation to be used (typically use a standard rep)

Goal is to be as fast as possible on the most common architectures, yet must also be very general

Request marshalling
Client (stub) builds a message containing arguments, indicates what procedure to invoke Due to need for generality, data representation a potentially costly issue! Performs a send I/O operation to send the message Performs a receive I/O operation to accept the reply Unpacks the reply from the reply message Returns result to the client program

Steps of a Remote Procedure Call

1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

Client procedure calls client stub in normal way Client stub builds message, calls local OS Client's OS sends message to remote OS Remote OS gives message to server stub Server stub unpacks parameters, calls server Server does work, returns result to the stub Server stub packs it in message, calls local OS Server's OS sends message to client's OS Client's OS gives message to client stub Stub unpacks result, returns to client

Client and Server Stubs

Principle of RPC between a client & server program.

Client/server programs do not know what stubs are doing Think they are calling a library function from the header file

Example (Sun RPC - ONC)

long square(long) example
client ren.eecis.udel.edu 11 result: 121

Need RPC specification file (square.x): defines procedure name, arguments & results Run rpcgen square.x: generates square.h, square_clnt.c, square_xdr.c, square_svc.c square_clnt.c & square_svc.c: Stub routines for client & server square_xdr.c: XDR (External Data Representation) code - takes care of data type conversions

RPC Specification File (square.x)

struct square_in { long arg1; }; struct square_out { long res1; }; IDL Interface Definition Language program SQUARE_PROG { version SQUARE_VERS { square_out SQUAREPROC(square_in) = 1; } = 1; // version # } = 0x321230000; // program #

// procedure #

Parameter Specification & Stub Generation

procedure

Corresponding message

Writing a Client & a Server

The steps in writing a client & a server in DCE RPC.

Costs in basic protocol?

Allocation and marshalling data into message (can reduce costs if you are certain client, server have identical data representations) Two system calls, one to send, one to receive, hence context switching Much copying all through the O/S: application to UDP, UDP to IP, IP to ethernet interface, and back up to application

Schroeder and Burrows*

Studied RPC performance in O/S kernel Suggested a series of major optimizations Resulted in performance improvements of about 10-fold for Xerox firefly workstation (from 10ms to below 1ms)

*Shroader, M and Burrows, Performance of Firefly RPC, 12th ACM Symposium On Operating Systems Principles, December 1989.

Typical optimizations?
Compile the stub inline to put arguments directly into message Two versions of stub; if (at bind time) sender and dest. found to have same data representations, use host-specific rep. Use a special send, then receive system call (requires O/S extension) Optimize the O/S kernel path itself to eliminate copying treat RPC as the most important task the kernel will do

Fancy argument passing

RPC is transparent for simple calls with a small amount of data passed
Transparent in the sense that the interface to the procedure is unchanged But exceptions thrown will include new exceptions associated with network

What about complex structures, pointers, big arrays? These will be very costly, and perhaps impractical to pass as arguments Most implementations limit size, types of RPC arguments. Very general systems less limited but much more costly.

Overcoming lost packets

client sends request server

Overcoming lost packets

client sends request Timeout! retransmit ack for request duplicate request: ignored server

Overcoming lost packets

client sends request Timeout! retransmit ack for request reply server

Overcoming lost packets

client sends request Timeout! retransmit ack for request reply ack for reply server

Costs in fault-tolerant version

Acks are expensive. Try and avoid them, e.g. if the reply will be sent quickly suppress the initial ack Retransmission is costly. Try and tune the delay to be optimal For big messages, send packets in bursts and ack a burst at a time, not one by one

Big messages
client sends request as a burst server

ack entire burst reply ack for reply

RPC semantics
Normally: client sends request, server executes, returns response What happens when server or network fails? Hang forever waiting for reply
Models regular procedure call

Time out and raise exception or report failure to client

What timer value to use?

Time out and retransmit request

What if client transmits, server executes, but then crashes before sending reply? How do we know if server acted on request?

RPC semantics
At most once: request is processed 0 or 1 times Exactly once: request is always processed 1 time At least once: request processed 1 or more times

... but exactly once is impossible because we cant distinguish packet loss from true failures! (did the net fail to deliver message or server crash before sending the ack that my paris ticket purchase was satisfied?) In both cases, RPC protocol simply times out.

Implementing at most/least once

Use a timer (clock) value and a unique id, plus sender address Server remembers recent ids and replies with same data if a request is repeated Also uses id to identify duplicates and reject them Very old requests detected and ignored by checking time
Assumes that the clocks are working In particular, requires synchronized clocks

RPC versus local procedure call

Restrictions on argument sizes and types New error cases:
Bind operation failed Request timed out Argument too large can occur if, e.g., a table grows

Costs may be very high ... so RPC is actually not very transparent!

RPC costs in case of local destination process

Often, the destination is right on the callers machine!
Caller builds message Issues send system call, blocks, context switch Message copied into kernel, then out to dest. Dest is blocked... wake it up, context switch Dest computes result Entire sequence repeated in reverse direction If scheduler is a process, context switch 5 times!

RPC example

Dest on same site O/S

Source does xyz(a, b, c)

RPC in normal case

Destination and O/S are blocked Dest on same site O/S

Source does xyz(a, b, c)

RPC in normal case

Source, dest both block. O/S runs its scheduler, copies message from source outqueue to dest in-queue Dest on same site O/S

Source does xyz(a, b, c)

RPC in normal case

Dest runs, copies in message Dest on same site O/S

Source does xyz(a, b, c)

Same sequence needed to return results

Important optimizations: LRPC*

Lightweight RPC (LRPC): for case of sender, dest on same machine (Bershad et. al.) Uses memory mapping to pass data Reuses same kernel thread to reduce context switching costs (user suspends and server wakes up on same kernel thread or stack) Single system call: send_rcv or rcv_send
*Bershad, B. Anderson, T., Lazowska, E. Levy, H. Lightweight Remote Procedure Call, ACM Transactions on Computer Systems, Vol 8, Issue 1, February 1990.

LRPC
O/S and dest initially are idle

Dest on same site O/S

Source does xyz(a, b, c)

LRPC
Control passes directly to dest

Dest on same site O/S

Source does xyz(a, b, c)

arguments directly visible through remapped memory

LRPC performance impact

On same platform, offers about a 10-fold improvement over a hand-optimized RPC implementation Does two memory remappings, no context switch Runs about 50 times faster than standard RPC by same vendor (at the time of the research) Semantics stronger: easy to ensure exactly once

Broad comments on RPC

RPC is not very transparent Failure handling is not evident at all: if an RPC times out, what should the developer do?
Reissuing the request only makes sense if there is another server available Anyhow, what if the request was finished but the reply was lost? Do it twice? Try to duplicate the lost reply?

Performance work is producing enormous gains: from the old 75ms RPC to modern-day RPC, research has led to a factor of 1000 performance improvement!

Contents of an RPC environment

Standards for data representation IDL files describing service specifications (procedures supported, arguments, etc). Stub compilers to transform spec into stub implementations for client and server Services to manage server directory, clock synchronization Tools for visualizing system state and managing servers and applications

Closely Related Topic

Multithreading is a common performanceenhancing technique Idea is that server is often idle while doing I/O for one client, so use extra threads to allow concurrent request processing In the limit, leads to database transactional concurrency model, but many nontransactional servers use threads for enhanced performance

Multithreading debate
Three major options:
Single-threaded server: only does one thing at a time, uses send/recv system calls and blocks while waiting
Non-blocking single-threaded server with bookkeeping: on an I/O for a request, saves state in pending activities table, starts on another request

Multi-threaded server: internally concurrent, each request spawns a new thread to handle it Upcalls: event dispatch loop does a procedure call for each incoming event, like for X11 or PCs running Windows.

Single threading: drawbacks

Applications can deadlock if a request cycle forms: Im waiting for you and you send me a request, which I cant handle Much of system may be idle waiting for replies to pending requests Harder to implement RPC protocol itself (need to use a timer interrupt to trigger acks, retransmission, which is awkward)

Multithreading
Idea is to support internal concurrency as if each process was really multiple processes that share one address space Thread scheduler uses timer interrupts and context switching to mimic a physical multiprocessor using the smaller number of CPUs actually available

Multithreaded RPC
Each incoming request is handled by spawning a new thread Designer must implement appropriate mutual exclusion to guard against race conditions and other concurrency problems Ideally, server is more active because it can process new requests while waiting for its own RPCs to complete on other pending requests

Negatives to multithreading
Users may have little experience with concurrency and will then make mistakes Concurrency bugs are very hard to find due to nonreproducible scheduling orders Reentrance & ordering issues can come as an undesired surprise different data update behaviors Threads need stacks hence consumption of memory can be very high Deadlock remains a risk, now associated with concurrency control Stacks for threads must be finite and can overflow, corrupting the address space

Threads: can spawn too many

SCHED

event

Threads: can spawn too many

Thread spawned, but blocks

SCHED

event

Threads: can spawn too many

SCHED

Eventually, application becomes bloated, begins to thrash. Performance drops and clients may think the server has failed

event

Upcall model
Common in windowing systems Each incoming event is encoded as a small descriptive data structure User registers event handling procedures Dispatch loop calls the procedures as new events arrive, waits for the call to finish, then dispatches a new event

Upcalls combined with threads

Some say perhaps the best model for RPC programming Each handler can be tagged: needs thread, or can be executed unthreaded Developer must still be very careful where threads are used

Recent RPC history

RPC was once touted as the transparent answer to distributed computing Today the protocol is very widely used ... but it isnt very transparent, and reliability issues can be a major problem Today the strongest interest is in Web Services and CORBA, which use RPC as the mechanism to implement object invocation