C NPv1

C++ Network Programming
Mastering Complexity with ACE & Patterns

Dr. Douglas C. Schmidt
d.schmidt@vanderbilt.edu www.cs.wustl.edu/~schmidt/tutorials-ace.html Professor of EECS Vanderbilt University Nashville, Tennessee
Motivation: Challenges of Networked Applications

Observation Building robust, efficient, & extensible concurrent & networked applications is hard e.g., we must address many complex topics that are less problematic for nonconcurrent, stand-alone applications Complexities in networked applications
Accidental Complexities Low-level APIs Poor debugging tools Algorithmic decomposition Continuous re-invention/discovery of core concepts & components Inherent Complexities Latency Reliability Load balancing Causal ordering Scheduling & synchronization Deadlock
Presentation Outline
Cover OO techniques & language features that enhance software quality
Patterns, which embody reusable software architectures & designs ACE wrapper facades, which encapsulate OS concurrency & network programming APIs OO language features, e.g., classes, dynamic binding & inheritance, parameterized types Presentation Organization 1. Background 2. Concurrent & network challenges & solution approaches 3. Patterns & wrapper facades in ACE + applications
3
The Layered Architecture of ACE

www.cs.wustl.edu/~schmidt/ACE.html
Features Open-source 200,000+ lines of C++ 40+ personyears of effort Ported to many OS platforms
Large open-source user community www.cs.wustl.edu/~schmidt/ACE-users.html 14
Commercial support by Riverace www.riverace.com/
Sidebar: Platforms Supported by ACE

ACE runs on a wide range of operating systems, including: PCs, e.g., Windows (all 32/64-bit versions), WinCE; Redhat, Debian, and SuSE Linux; & Macintosh OS X;
Most versions of UNIX, e.g., SunOS 4.x and Solaris, SGI IRIX, HPUX, Digital UNIX (Compaq Tru64), AIX, DG/UX, SCO OpenServer, UnixWare, NetBSD, & FreeBSD;
Real-time operating systems, e.g., VxWorks, OS/9, Chorus, LynxOS, Pharlap TNT, QNX Neutrino and RTP, RTEMS, & pSoS; Large enterprise systems, e.g., OpenVMS, MVS OpenEdition, Tandem NonStop-UX, & Cray UNICOS ACE can be used with all of the major C++ compilers on these platforms
The ACE Web site at http://www.cs.wustl.edu/~schmidt/ACE.html contains a complete, up-to-date list of platforms, along with instructions for downloading & building ACE
15
Key Capabilities Provided by ACE

Service Access & Control Event Handling & IPC
Concurrency
Synchronization
16
The Pattern Language for ACE

Pattern Benefits Preserve crucial design information used by applications & middleware frameworks & components Facilitate reuse of proven software designs & architectures Guide design choices for application developers
17
The Frameworks in ACE
ACE Framework Reactor & Proactor Service Configurator Task Acceptor-Connector
Inversion of Control Calls back to application-supplied event handlers to perform processing when events occur synchronously & asynchronously Calls back to application-supplied service objects to initialize, suspend, resume, & finalize them Calls back to an application-supplied hook method to perform processing in one or more threads of control Calls back to service handlers to initialize them after they are connected
Streams
20
Calls back to initialize & finalize tasks when they are pushed & popped from a stream
Networked Logging Service Example

Key Participants Client application processes Generate log records Server logging daemon Receive, process, & store log records C++ code for all logging service examples are in
ACE_ROOT/examples/ C++NPv1/ ACE_ROOT/examples/ C++NPv2/
The logging server example in C++NPv2 is more sophisticated than the one in C++NPv1 Theres an extra daemon involved
26
Patterns in the Networked Logging Service

Half-Sync/ Half-Async Leader/ Followers Monitor Object
Active Object
Reactor
Pipes & Filters AcceptorConnector Component Configurator Proactor Wrapper Facade
Strategized Locking
Scoped Locking
Thread-safe Interface
27
ACE Basics: Logging

ACEs logging facility usually best for diagnostics Can customize logging sinks Filterable logging severities Portable printf()-like format directives (thread/process ID, date/time, types) Serializes output across multiple threads ACE propagates settings to threads created via ACE Can log across a network ACE_Log_Msg class; use thread-specific singleton most of the time, via ACE_LOG_MSG macro Macros encapsulate most usage. Most common: ACE_DEBUG ((severity, format [, args])); ACE_ERROR[_RETURN] ((severity, format [,args])[, return-value]); See ACE Programmers Guide (APG) tables 3.1 (severities), 3.2 (directives), 3.3 (macros)
28
ACE Logging Usage

The ACE logging API is similar to printf(), e.g.: ACE_ERROR ((LM_ERROR, "(%t) fork failed")); generates: Oct 31 14:50:13 1992@ics.uci.edu@2766@LM_ERROR@client::(4) fork failed and ACE_DEBUG ((LM_DEBUG, "(%t) sending to server %s", host)); generates: Oct 31 14:50:28 1992@ics.uci.edu@1832@LM_DEBUG@drwho::(6) sending to server tango
Format %l %N %n %P %p Action Displays the line number where the error occurred Displays the file name where the error occurred Displays the name of the program Displays the current process ID Takes a const char * argument and displays it and the error string corresponding to errno (similar to perror())
%T
29
Displays the current time
Logging Severities
You can control which severities are seen at run time Two masks determine whether a message is displayed: Process-wide mask (defaults to all severities enabled) Per-thread mask (defaults to all severities disabled) If logged severity is enabled in either mask, message is displayed Set process/instance mask with:
ACE_Log_Msg::priority_mask (u_long mask, MASK_TYPE which);
MASK_TYPE is ACE_Log_Msg::PROCESS or ACE_Log_Msg::THREAD.
Since default is to enable all severities process-wide, all severities are logged in all threads unless you change it Per-thread mask initializer can be adjusted (default is all severities disabled):
ACE_Log_Msg::disable_debug_messages (); ACE_Log_Msg::enable_debug_messages();
Any set of severities can be specified (ORd together) Note that these methods set and clear a (set of) bits instead of replacing the mask, as priority_mask() does
30
Logging Severities Example

To allow threads to decide their own logging, the desired severities must be: Disabled at process level & enabled in the thread(s) to display them. e.g.,
ACE_LOG_MSG->priority_mask (0, ACE_Log_Msg::PROCESS); ACE_Log_Msg::enable_debug_messages ();
ACE_Thread_Manager::instance ()->spawn (service);

ACE_Log_Msg::disable_debug_messages (); ACE_Thread_Manager::instance ()->spawn_n (3, worker);
LM_DEBUG severity (only) logged in service thread LM_DEBUG severity (and all others) not logged in worker threads Note that enable_debug_messages() & disable_debug_messages() are static methods
31
Redirect Logging to a File

Default logging sink is stderr. Redirect to a file by setting the OSTREAM flag and assigning a stream. Can set the flag in two ways:
ACE_Log_Msg::open (const ACE_TCHAR *prog_name, u_long options_flags = ACE_Log_Msg::STDERR, const ACE_TCHAR *logger_key = 0);
ACE_Log_Msg::set_flags (u_long flags);
Assign a stream: ACE_Log_Msg::msg_ostream (ACE_OSTREAM_TYPE *); (Optional 2nd arg to tell ACE_Log_Msg to delete the ostream)
ACE_OSTREAM_TYPE is ofstream where supported, else FILE* To also stop output to stderr, use open() without STDERR flag, or ACE_Log_Msg::clr_flags (STDERR)
32
Redirect Logging to Syslog

Redirected log output to ACE_Log_Msg::SYSLOG goes to: Windows NT4 and up: systems Event Log UNIX/Linux: syslog facility (uses LOG_USER syslog facility) Cant set this with set_flags/clr_flags; must open. For example: ACE_LOG_MSG->open (argv[0], ACE_Log_Msg::SYSLOG, ACE_TEXT (syslogTest)); Windows: 3rd arg, if supplied, replaces 1st as program name in event log To turn it off, call open() again with different flag(s). This seems odd, but youre effectively resetting the logging think of it as reopen().
33
Logging Callbacks
Logging callbacks are useful for adding special processing or filtering to log output Derive a class from ACE_Log_Msg_Callback & reimplement: virtual void log (ACE_Log_Record &log_record); Use ACE_Log_Msg::msg_callback() to register callback Also call ACE_Log_Msg::set_flags() to add ACE_Log_Msg::MSG_CALLBACK flag Beware Callback registration is specific to each ACE_Log_Msg instance Callbacks are not inherited when new threads are created
34
Useful Logging Flags

There are some other ACE_Log_Msg flags that add useful functionality to ACEs logging: VERBOSE: Prepends program name, timestamp, host name, process ID, and message priority to each message VERBOSE_LITE: Prepends timestamp and message priority to each message (this is what ACE test suite uses) SILENT: Dont display any messages of any severity LOGGER: Write messages to the local client logger deamon
35
Tracing
ACEs tracing facility logs function/method entry & exit Uses logging with severity LM_TRACE, so output can be selectively disabled Just put ACE_TRACE macro in the function: #include ace/Log_Msg.h void foo (void) { ACE_TRACE (foo); // do stuff } Says: (1024) Calling foo in file test.cpp on line 8 (1024) Leaving foo Clever indenting by call depth makes output easier to read Huge amount of output, so tracing no-opd out by default; rebuild with config.h having: #define ACE_NTRACE 0
36
Networked Logging Service Example

Key Participants Client application processes Generate log records Server logging daemon Receive, process, & store log records C++ code for all logging service examples are in
ACE_ROOT/examples/ C++NPv1/ ACE_ROOT/examples/ C++NPv2/
Well develop architecture similar to ACEs, but not same implementation.
37
Network Daemon Design Dimensions
Communication dimensions address the rules, form, & level of abstraction that networked applications use to interact Concurrency dimensions address the policies & mechanisms governing the proper use of processes & threads to represent multiple service instances, as well as how each service instance may use multiple threads internally
38
Service dimensions address key properties of a networked application service, such as the duration & structure of each service instance Configuration dimensions address how networked services are identified & the time at which they are bound together to form complete applications
Communication Design Dimensions
Communication is fundamental to networked application design The next three slides present a domain analysis of communication design dimensions, which address the rules, form, and levels of abstraction that networked applications use to interact with each other We cover the following communication design dimensions: Connectionless versus connection-oriented protocols Synchronous versus asynchronous message exchange Message-passing versus shared memory
39
Connectionless vs. Connection-oriented Protocols

A protocol is a set of rules that specify how control & data information is exchanged between communicating entities
SYN SYN/ACK ACK
Connector
Acceptor
3-way handshake in TCP/IP

Connection-oriented protocols Connectionless protocols provide
provide a reliable, sequences, nonduplicated delivery service, which is useful for applications that cant tolerate data loss Examples include TCP & ATM
a message-oriented service in which each message can be routed and delivered independently Examples include UDP & IP
Connection-oriented applications must address two additional design issues: Data framing strategies, e.g., bytestream vs. message-oriented Connection multiplexing (muxing) strategies, e.g., multiplexed vs. nonmultiplexed
40
Alternative Connection Muxing Strategies
In multiplexed connections all client requests emanating from threads in a single process pass through one TCP connection to a server process Pros: Conserves OS communication resources, such as socket handles and connection control blocks Cons: harder to program, less efficient, & less deterministic 41
In nonmultiplexed connections each client uses a different connection to communicate with a peer service Pros: Finer control of communication priorities & low synchronization overhead since additional locks aren't needed
Cons: use more OS resources, & therefore may not scale well in certain environments
Sync vs. Async Message Exchange

Asynchronous request/response protocols stream requests from client to server without waiting for responses synchronously Multiple client requests can be transmitted before any responses arrive from a server Synchronous request/response protocols are the simplest form to implement Requests & responses are exchanged in a lock-step sequence. Each request must receive a response synchronously before the next is sent
42
These protocols therefore often require a strategy for detecting lost or failed requests & resending them later
Message Passing vs. Shared Memory
Message passing exchanges data explicitly via the IPC mechanisms
Shared memory allows multiple processes on the same or different Application developers generally define the hosts to access & exchange data as though it were local to the address protocol for exchanging the data, e.g.: space of each process Format & content of the data Applications using native OS shared Number of possible participants in each memory mechanisms must define exchange (e.g., point-to-point unicast), how to locate & map the shared multicast, or broadcast) memory region(s) & the data How participants begin, conduct, & end structures that are placed in shared a message-passing session memory
43
Sidebar: C++ Objects & Shared Memory

Allocating a C++ Object in shared Memory
void *obj_buf = // Get a pointer to location in shared memory ABC *abc = new (obj_buf) ABC; // Use C++ placement new operator
General responsibilities using placement new operator
Pointer passed to placement new operator must point to a memory region that is big enough & is aligned properly for the object type being created
The placed object must be destroyed by explicitly calling the destructor Pitfalls initializing C++ objects with virtual functions in shared memory
The shared memory region may reside at a different virtual memory location in each process that maps the shared memory
The C++ compiler/linker need not locate the vtable at the same address in different processes that use the shared memory
ACE wrapper faade classes that can be initialized in shared memory must therefore be concrete data types
i.e., classes with only non-virtual methods
44
Overview of the Socket API (1/2)

Sockets are the most common network programming API available on operating system platforms Originally developed in BSD Unix as a C language API to TCP/IP protocol suite The Socket API has approximately two dozen functions classified in five categories Socket is a handle created by the OS that associates it with an end point of a communication channel
45
A socket can be bound to a local or remote address In Unix, socket handles & I/O handles can be used interchangeably in most cases, but this is not the case for Windows
Overview of the Socket API (2/2)

Local context management
Connection establishment & termination
Data transfer mechanisms
Options management Network addressing

46
Taxonomy of Socket Dimensions

The Socket API can be decomposed into the following dimensions: Type of communication service
e.g., streams versus datagrams versus connected datagrams
Communication & connection role

e.g., clients often initiate connections actively, whereas servers often accept them passively
Communication domain
e.g., local host only versus local or remote host
47
Limitations with the Socket APIs (1/2)

Poorly structured, non-uniform, & non-portable
API is linear rather than hierarchical

i.e., the API is not structured according to the different phases of connection lifecycle management and the roles played by the participants
No consistency among the names Non-portable & error-prone

Function names: read() & write() used for any I/O handle on Unix but Windows needs ReadFile() & WriteFile() Function semantics: different behavior of same function on different OS e.g., accept () can take NULL client address parameter on Unix/Windows, but will crash on some operating systems, such as VxWorks Socket handle representations: different platforms represent sockets differently e.g., Unix uses unsigned integers whereas Windows uses pointers Header files: Different platforms use different names for header files for the socket API
48
Limitations with the Socket APIs (2/2)

Lack of type safety
I/O handles are not amenable to strong type checking at compile time e.g., no type distinction between a socket used for passive listening & a socket used for data transfer
Steep learning curve due to complex semantics

Multiple protocol families & address families
Options for infrequently used features such as broadcasting, async I/O, non blocking I/O, urgent data delivery Communication optimizations such as scatter-read & gather-write Different communication and connection roles, such as active & passive connection establishment, & data transfer
Too many low-level details

Forgetting to use the network byte order before data transfer Possibility of missing a function, such as listen() Possibility of mismatch between protocol & address families Forgetting to initialize underlying C structures e.g., sockaddr Using a wrong socket for a given role
49
Example of Socket API Limitations (1/3)

1 #include <sys/types.h> 2 #include <sys/socket.h> Possible differences in header file names
3
4 const int PORT_NUM = 10000; 5 6 int echo_server () 7 { 8 9 struct sockaddr_in addr; int addr_len; Forgot to initialize to sizeof (sockaddr_in)
10
11 12
char buf[BUFSIZ];
int n_handle; Use of non-portable handle type
// Create the local endpoint.
50

13 14 15 16 17 18 int s_handle = socket (PF_UNIX, SOCK_DGRAM, 0); if (s_handle == -1) return -1;
Use of non-portable return value

// Set up address information where server listens. Protocol and address family addr.sin_family = AF_INET; mismatch addr.sin_port = PORT_NUM;
Wrong byte order
19
20 21 22 23 24
51
addr.sin_addr.addr = INADDR_ANY; Unused structure members not zeroed out

if (bind (s_handle, (struct sockaddr *) &addr, sizeof addr) == -1) return -1; Missed call to listen()

25 26 27 28 29 30 int n; // Create a new communication endpoint. if (n_handle = accept (s_handle, (struct sockaddr *) &addr, &addr_len) != -1) {
SOCK_DGRAM handle illegal here
while ((n = read (s_handle, buf, sizeof buf)) > 0) write (n_handle, buf, n); Reading from wrong handle
31
32 33 34 35 } } return 0;
No guarantee that n bytes will be written close (n_handle);
52
ACE Socket Wrapper Faade Classes

ACE defines a set of C++ classes that address the limitations with the Socket API Enhance type-safety Ensure portability Simplify common use cases Building blocks for higher-level abstractions These classes are designed in accordance with the Wrapper Facade design pattern
53
The Wrapper Faade Pattern (1/2)

Context
Networked applications must manage a variety of OS services, including processes, threads, socket connections, virtual memory, & files OS platforms provide low-level APIs written in C to access these services
Applications
Problem
The diversity of hardware & operating systems makes it hard to build portable & robust networked application software Programming directly to low-level OS APIs is tedious, error-prone, & non54 portable
Solaris VxWorks Win2K Linux LynxOS
The Wrapper Faade Pattern (2/2)

Solution
Apply the Wrapper Facade design pattern (P2) to avoid accessing low-level operating system APIs directly
Wrapper Facade
calls methods calls calls
API FunctionA()
Application
data method1() methodN()
API FunctionB()
API FunctionC()
calls
This pattern encapsulates data & functions provided by existing non-OO APIs within more concise, robust, portable, maintainable, & cohesive OO class interfaces
55
void method1(){ functionA(); functionB(); }
void methodN(){ functionA(); }
: Application
: Wrapper Facade
method()
: APIFunctionA
: APIFunctionB
functionA() functionB()
ACE Socket Wrapper Faades Taxonomy

The structure of the ACE Socket wrapper facades reflects the domain of networked IPC properties The ACE Socket wrapper faade classes provide the following capabilities: ACE_SOCK_* classes encapsulate Internet-domain Socket API functionality ACE_LSOCK_* classes encapsulate UNIX-domain Socket API functionality
56
ACE also has wrapper facades for datagrams e.g., unicast, multicast, broadcast
Roles in the ACE Socket Wrapper Facade
57
The active connection role (ACE_SOCK_Connector) is played by a peer application that initiates a connection to a remote peer The passive connection role (ACE_SOCK_Acceptor) is played by a peer application that accepts a connection from a remote peer & The communication role (ACE_SOCK_Stream) is played by both peer applications to exchange data after they are connected
ACE Socket Addressing Classes (1/2)

Motivation Network addressing is a trouble spot in the Socket API
To minimize the complexity of these low-level details, ACE defines a hierarchy of classes that provide a uniform interface for all ACE network addressing objects
58
ACE Socket Addressing Classes (2/2)
Class Capabilities The ACE_Addr class is the root of the ACE network addressing hierarchy The ACE_INET_Addr class represents TCP/IP & UDP/IP addressing information This class eliminates many subtle sources of accidental complexity
59
ACE I/O Handle Classes (1/2)

Motivation Low-level C I/O handle types are tedious, error-prone, & non-portable
Even the ACE_HANDLE typedef is still not sufficiently object-oriented & typesafe
int buggy_echo_server (u_short port_num) { sockaddr_in s_addr; int acceptor = socket (PF_UNIX, SOCK_DGRAM, 0); int is not portable to Windows s_addr.sin_family = AF_INET; s_addr.sin_port = port_num; s_addr.sin_addr.s_addr = INADDR_ANY; bind (acceptor, (sockaddr *) &s_addr, sizeof s_addr); int handle = accept (acceptor, 0, 0); for (;;) { char buf[BUFSIZ]; ssize_t n = read (acceptor, buf, sizeof buf); if (n <= 0) break; Reading from wrong handle write (handle, buf, n); } }
60
ACE I/O Handle Classes (2/2)

Class Capabilities ACE_IPC_SAP is the root of the ACE hierarchy of IPC wrapper facades It provides basic I/O handle manipulation capabilities to other ACE IPC wrapper facades ACE_SOCK is the root of the ACE Socket wrapper facades & it provides methods to
Create & destroy socket handles Obtain the network addresses of local & remote peers
Set/get socket options, such as socket queue sizes, Enable broadcast/multicast communication
61
Disable Nagles algorithm
The ACE_SOCK_Connector Class

Motivation There is a confusing asymmetry in the Socket API between (1) connection roles & (2) socket modes e.g., an application may accidentally call recv() or send() on a data-mode socket handle before it's connected This problem can't be detected until run time since C socket handles are weakly-typed
int buggy_echo_client (u_short port_num, const char *s) { int handle = socket (PF_UNIX, SOCK_DGRAM, 0); write (handle, s, strlen (s) + 1); sockaddr_in s_addr; Operations called in memset (&s_addr, 0, sizeof s_addr); wrong order s_addr.sin_family = AF_INET; s_addr.sin_port = htons (port_num); connect (handle, (sockaddr *) &s_addr, sizeof s_addr); }
62
The ACE_SOCK_Connector Class

Class Capabilities ACE_SOCK_Connector is factory that establishes a new endpoint of communication actively & provides capabilities to Initiate a connection with a peer acceptor & then to initialize an ACE_SOCK_Stream object after the connection is established Initiate connections in either a blocking, nonblocking, or timed manner Use C++ traits to support generic programming techniques that enable wholesale replacement of IPC functionality
63
Sidebar: Traits for ACE Wrapper Facades (1/2)

ACE uses the C++ generic programming idiom to define & combine a set of characteristics to alter the behavior of a template class In C++, the typedef & typename language feature is used to define a trait A trait provides a convenient way to associate related types, values, & functions with template parameter type without requiring that they be defined as members of the type Traits are used extensively in the C++ Standard Template Library (STL)
64
Sidebar: Traits for ACE Wrapper Facades (2/2)

ACE Socket wrapper facades use traits to define the following associations PEER_ADDR this trait defines the ACE_INET_Addr class associated with the ACE Socket Wrapper Faade PEER_STREAM this trait defines the ACE_SOCK_Stream data transfer class associated with the ACE_SOCK_Acceptor & ACE_SOCK_Connector factories
class ACE_SOCK_Connector { public: typedef ACE_INET_Addr PEER_ADDR; typedef ACE_SOCK_Stream PEER_STREAM; // ... class ACE_TLI_Connector { public: typedef ACE_INET_Addr PEER_ADDR; typedef ACE_TLI_Stream PEER_STREAM; // ...
65
Using the ACE_SOCK_Connector (1/3)

This example shows how the ACE_SOCK_Connector can be used to connect a client application to a Web server
int main (int argc, char *argv[]) { const char *pathname = argc > 1 ? argv[1] : /index.html"; const char *server_hostname = Instantiate the connector, argc > 2 data transfer, & address ? argv[2] : www.dre.vanderbilt.edu"; objects typedef ACE_SOCK_Connector CONNECTOR; CONNECTOR connector; CONNECTOR::PEER_STREAM peer; CONNECTOR::PEER_ADDR peer_addr; if (peer_addr.set (80, server_hostname) == -1) Block until connection return 1; established or else if (connector.connect (peer, peer_addr) == -1) connection request failure return 1;
66

// Designate a nonblocking connect. Perform a non-blocking if (connector.connect (peer, connect peer_addr, &ACE_Time_Value::zero) == -1) { if (errno == EWOULDBLOCK) { // Do some other work ... // Now, try to complete connection establishment, // but don't block if it isn't complete yet. if (connector.complete (peer, 0, If connection not &ACE_Time_Value::zero) == -1) established, do other work & try again without blocking
// Designate a timed connect. ACE_Time_Value timeout (10); // 10 second timeout. if (connector.connect (peer, Perform a timed connect peer_addr, e.g., 10 seconds in this &timeout) == -1) { case if (errno == ETIME) { // Timeout, do something else 67

The ACE_SOCK_Connector can be passed the following values to control its timeout behavior
68
The ACE_SOCK_Stream Class (1/2)

Motivation Developers can misuse sockets in ways that can't be detected during compilation An ACE_SOCK_Stream object can't be used in any role other than data transfer without violating its (statically type-checked) interface
int buggy_echo_server (u_short port_num) { sockaddr_in s_addr; int acceptor = socket (PF_UNIX, SOCK_DGRAM, 0); s_addr.sin_family = AF_INET; s_addr.sin_port = port_num; s_addr.sin_addr.s_addr = INADDR_ANY; bind (acceptor, (sockaddr *) &s_addr, sizeof s_addr); int handle = accept (acceptor, 0, 0); for (;;) { char buf[BUFSIZ]; ssize_t n = read (acceptor, buf, sizeof buf); if (n <= 0) break; Reading from wrong handle write (handle, buf, n); } }
69
The ACE_SOCK_Stream Class (2/2)

Class Capabilities Encapsulates data transfer mechanisms supported by data-mode sockets to provide the following capabilities: Support for sending & receiving up to n bytes or exactly n bytes Support for scatter-read, which populate multiple callersupplied buffers instead of a single contiguous buffer Support for ``gather-write'' operations, which transmit the contents of multiple noncontiguous data buffers in a single operation Support for blocking, nonblocking, & timed I/O operations Support for generic programming techniques that enable the wholesale replacement of functionality via C++ parameterized types
70
Using the ACE_SOCK_Stream (1/2)

This example shows how an ACE_SOCK_Stream can be used to send & receive data to & from a Web server
// ...Connection code from example in Section 3.5 omitted... char buf[BUFSIZ]; Initialize the iovec iovec iov[3]; vector for scatter-read & gather-write I/O iov[0].iov_base = (char *) "GET "; iov[0].iov_len = 4; // Length of "GET ". iov[1].iov_base = (char *) pathname; iov[1].iov_len = strlen (pathname); iov[2].iov_base = (char *) " HTTP/1.0\r\n\r\n"; iov[2].iov_len = 13; // Length of " HTTP/1.0\r\n\r\n"; if (peer.sendv_n (iov, 3) == -1) return 1;
Perform blocking gather-
write on ACE_SOCK_Stream
for (ssize_t n; (n = peer.recv (buf, sizeof buf)) > 0; ) ACE::write_n (ACE_STDOUT, buf, n); return peer.close () == -1 ? 1 : 0;
Perform blocking read on
}
71
ACE_SOCK_Stream
Using the ACE_SOCK_Stream (2/2)

Blocking & non-blocking I/O semantics can be controlled via the ACE_SOCK_STREAM enable() & disable() methods, e.g., peer.enable (ACE_NONBLOCK); // enables non blocking peer.disable (ACE_NONBLOCK); // disable non blocking If the I/O operation blocks, it returns a -1 & errno is set to EWOULDBLOCK I/O operations can involve timeouts, e.g., ACE_Time_Value timeout (10); // 10 second timeout If (peer.sendv_n (iov, 3, &timeout) == -1) { // check if errno is set to ETIME, // which indicates a timeout } // similarly use timeout for receiving data
72
Sidebar: Working with (& Around) Nagles Algorithm

Nagles Algorithm Problem: Need to tackle the send-side silly window syndrome, where small data payloads, such as a keystroke, result in transmissions of large packets & causing unnecessary waste of network resources & congestion Solution: The OS kernel buffers a # of small-sized application messages & concatenates them into a larger size packet that can then be transmitted Consequences: Although network congestion is minimized, it can lead to higher & unpredictable latencies, as well as lower throughput Controlling Nagles Algorithm via ACE Use the set_option() method of the ACE_SOCK class e.g., int nodelay = 1; // Disable Nagles algorithm ACE_SOCK_Stream option_setter (handle); if (-1 == option_setter.set_option (ACE_IPPROTO_TCP, TCP_NODELAY, &nodelay, sizeof (nodelay))) ...
73
The ACE_SOCK_Acceptor Class (1/2)

Motivation The C functions in the Socket API are weakly typed, which makes it easy to apply them incorrectly in ways that cant be detected until run-time The ACE_SOCK_Acceptor class ensures type errors are detected at compile-time
int buggy_echo_server (u_short port_num) { sockaddr_in s_addr; int acceptor = socket (PF_UNIX, SOCK_DGRAM, 0); s_addr.sin_family = AF_INET; s_addr.sin_port = port_num; s_addr.sin_addr.s_addr = INADDR_ANY; bind (acceptor, (sockaddr *) &s_addr, sizeof s_addr); int handle = accept (acceptor, 0, 0); for (;;) { char buf[BUFSIZ]; ssize_t n = read (acceptor, buf, sizeof buf); if (n <= 0) break; Reading from wrong handle write (handle, buf, n); } }
74
The ACE_SOCK_Acceptor Class (2/2)

Class Capabilities This class is a factory that establishes a new endpoint of communication passively & provides the following capabilities: It accepts a connection from a peer connector & then initializes an ACE_SOCK_Stream object after the connection is established Connections can be accepted in either a blocking, nonblocking, or timed manner C++ traits are used to support generic programming techniques that enable the wholesale replacement of functionality via C++ parameterized types
75
Using the ACE_SOCK_Acceptor

This example shows how an ACE_SOCK_Acceptor & ACE_SOCK_Stream can be used to accept connections & send/receive data to/from a web client extern char *get_url_pathname (ACE_SOCK_Stream *); int main () Instantiate the acceptor, data transfer, & address objects { ACE_INET_Addr server_addr; Initialize a passive ACE_SOCK_Acceptor acceptor; mode endpoint to ACE_SOCK_Stream peer; listen for connections on port 80 if (server_addr.set (80) == -1) return 1; if (acceptor.open (server_addr) == -1) return 1; Accept a new connection for (;;) { if (acceptor.accept (peer) == -1) return 1; peer.disable (ACE_NONBLOCK); // Ensure blocking <send_n>. ACE_Auto_Array_Ptr<char *> pathname (get_url_pathname (peer)); ACE_Mem_Map mapped_file (pathname.get ()); Send the requested data if (peer.send_n (mapped_file.addr (), mapped_file.size ()) == -1) return 1; peer.close (); Close the connection to the sender
Stop receiving any
}
76 }
return acceptor.close () == -1 ? 1 : 0;
connections
Sidebar: The ACE_Mem_Map Class

Memory Mapped Files ACE_Mem_Map Class
Many modern operating systems provide a mechanism for mapping a files contents directly into a processs virtual address space
This memory-mapped file mechanism can be read from or written to directly by referencing the virtual memory e.g., via pointers instead of using less efficient I/O functions
A wrapper faade that encapsulates the memory mapped file system mechanisms on different operating systems
Relieves application developers from having to manually perform bookkeeping tasks
The file manager defers all read/write operations to the virtual memory manager
Contents of memory mapped files can be shared by multiple processes on the same machine It can also be used to provide a persistent backing store 77
e.g., explicitly opening files or determining their lengths The ACE_Mem_Map class offers multiple constructors with several signature variants
The ACE_Message_Block Class (1/2)

MESSAGES BUFFERED FOR TRANSMISSION MESSAGES IN TRANSIT MESSAGES BUFFERED AWAITING PROCESSING
Motivation Many networked applications require a means to manipulate messages efficiently, e.g.: Storing messages in buffers as they are received from the network or from other processes Adding/removing headers/trailers from messages as they pass through a user-level protocol stack Fragmenting/reassembling messages to fit into network MTUs Storing messages in buffers for transmission or retransmission Reordering messages that were received out-of-sequence 78
The ACE_Message_Block Class (2/2)

Class Capabilities This class is a composite that enables efficient manipulation of messages via the following operations: Each ACE_Message_Block contains a pointer to a reference-counted ACE_Data_Block which in turn points to the actual data associated with a message It allows multiple messages to be chained together into a composite message It allows multiple messages to be joined together to form an ACE_Message_Queue It treats synchronization & memory management properties as aspects
79
Two Kinds of Message Blocks
Simple messages contain a Composite messages contain multiple one ACE_Message_Block ACE_Message_Blocks These blocks are linked together in accordance with An ACE_Message_Block the Composite pattern points to an Composite messages often consist of a control ACE_Data_Block message that contains bookkeeping information An ACE_Data_Block e.g., destination addresses, followed by one or points to the actual data more data messages that contain the actual payload contents of the message ACE_Data_Blocks can be referenced counted
80
Using the ACE_Message_Block (1/2)

The following program reads all data from standard input into a singly linked list of dynamically allocated ACE_Message_Blocks These ACE_Message_Blocks are chained together by their continuation pointers
Allocate an ACE_Message_Block int main (int argc, char *argv[]) whose payload is of size BUFSIZ { ACE_Message_Block *head = new ACE_Message_Block (BUFSIZ); ACE_Message_Block *mblk = head; Read data from standard input into the message block starting at write pointer (wr_ptr ()) for (;;) { ssize_t nbytes = ACE::read_n (ACE_STDIN, mblk->wr_ptr (), mblk->size ()); if (nbytes <= 0) break; // Break out at EOF or error.
mblk->wr_ptr (nbytes);
81
Advance write pointer by the number
of bytes read to end of buffer
Using the ACE_Message_Block (2/2)

Allocate a new ACE_Message_Block of size BUFSIZ &
chain it to the previous one at the end of the list mblk->cont (new ACE_Message_Block (BUFSIZ)); mblk = mblk->cont (); Advance mblk to point to the newly } allocated ACE_Message_Block
// Print the contents of the list to the standard output. for (mblk = head; mblk != 0; mblk = mblk->cont ()) ACE::write_n (ACE_STDOUT, mblk->rd_ptr (), mblk->length ());
For every message block, print mblk->length() amount
of contents starting at the read pointer (rd_ptr ())

Can also use ACE::write_n (head) to write entire
chain head->release (); // Release all the memory in the chain. return 0; }82
ACE CDR Streams

Motivation Networked applications that send/receive messages often require support for
Linearization
To handle the conversion of richly typed data to/from raw memory buffers
(De)marshaling
To interoperate with heterogeneous compiler alignments & hardware instructions with different byte-orders
The ACE_OutputCDR & ACE_InputCDR classes provide a highly optimized, portable, & convenient means to marshal & demarshal data using the standard CORBA Common Data Representation (CDR) ACE_OutputCDR creates a CDR buffer from a data structure (marshaling) 83 ACE_InputCDR extracts data from a CDR buffer (demarshaling)
The ACE_OutputCDR & ACE_InputCDR Classes

Class Capabilities ACE_OutputCDR & ACE_InputCDR support the following features: They provide operations to (de)marshal the following types: Primitive types, e.g., booleans; 16-, 32-, & 64-bit integers; 8-bit octets; single & double precision floating point numbers; characters; & strings Arrays of primitive types The insertion (<<) and extraction (>>) operators can marshal & demarshal primitive types, using the same syntax as the C++ iostream components ACE_Message_Block chains are used internally to minimize mem copies They take advantage of CORBA CDR alignment & byte-ordering rules to avoid memory copying & byte-swapping operations, respectively They provide optimized byte swapping code that uses inline assembly language instructions for common hardware platforms (such as Intel x86) & standard hton*()& ntoh*() macros/functions on other platforms
They support zero copy marshaling & demarshaling of octet buffers Users can define custom character set translators for platforms that do not 84 use ASCII or Unicode as their native character sets
Sidebar: Log Record Message Structure

ACE_Log_Record is a type that ACE uses internally to keep track of the fields in a log record
class ACE_Log_Record { private: ACE_UINT type_; ACE_UINT pid_; ACE_Time_Value timestamp_; char msg_data_[ACE_MAXLOGMSGLEN]; public: ACE_UINT type () const; ACE_UINT pid () const; const ACE_Time_Value timestamp () const; const char *msg_data () const; };
85
This example uses a 8-byte, CDR encoded header followed by the payload Header includes byte order, payload length, & other fields
Using ACE_OutputCDR
We show the ACE CDR insertion (operator<<) & extraction (operator>>) operators for ACE_Log_Record that's used by client application & logging server
int operator<< (ACE_OutputCDR &cdr, const ACE_Log_Record &log_record) { size_t msglen = log_record.msg_data_len (); // Insert each <log_record> field into the output CDR stream. cdr << ACE_CDR::Long (log_record.type ()); cdr << ACE_CDR::Long (log_record.pid ()); cdr << ACE_CDR::Long (log_record.time_stamp ().sec ()); cdr << ACE_CDR::Long (log_record.time_stamp ().usec ()); cdr << ACE_CDR::ULong (msglen); cdr.write_char_array (log_record.msg_data (), msglen); return cdr.good_bit (); }
86
After marshaling all the fields of the log record into the CDR stream, return the success/failure status
Using ACE_InputCDR
int operator>> (ACE_InputCDR &cdr, ACE_Log_Record &log_record) { ACE_CDR::Long type; Temporaries used during ACE_CDR::Long pid; demarshaling (not always ACE_CDR::Long sec, usec; necessary) ACE_CDR::ULong buffer_len; // Extract each field from input CDR stream into <log_record>. if ((cdr >> type) && (cdr >> pid) && (cdr >> sec) && (cdr >> usec) && (cdr >> buffer_len)) { ACE_TCHAR log_msg[ACE_Log_Record::MAXLOGMSGLEN + 1]; log_record.type (type); log_record.pid (pid); log_record.time_stamp (ACE_Time_Value (sec, usec)); cdr.read_char_array (log_msg, buffer_len); log_msg[buffer_len] = '\0'; log_record.msg_data (log_msg); } return cdr.good_bit (); After demarshaling all the fields of the log record from the CDR stream, return the success/failure } status
87
Implementing the Client Application (1/6)

The following client application illustrates how to use the ACE C++ Socket wrapper facades & CDR streams to establish connections, marshal log records, & send the data to our logging server This example behaves as follows: 1.Reads lines from standard input 2.Sends each line to the logging server in a separate log record & 3.Stops when it reads EOF from standard input
88
class Logging_Client { Header file: Logging_Client.h public: // Send <log_record> to the server. int send (const ACE_Log_Record &log_record); // Accessor method. ACE_SOCK_Stream &peer () { return logging_peer_; } // Close the connection to the server. ~Logging_Client () { logging_peer_.close (); } private: ACE_SOCK_Stream logging_peer_; // Connected to server. };

The Logging_Client::send() method behaves as follows: 1.Computes the size of the payload (lines 2 8)
2.Marshals the header & data into an output CDR (lines 10 16) &
3.Sends it to the logging server (lines 18 24)
1 int Logging_Client::send (const ACE_Log_Record &log_record) { 2 const size_t max_payload_size = 3 4 // type() 4 + 8 // timestamp 5 + 4 // process id 6 + 4 // data length 7 + ACE_Log_Record::ACE_MAXLOGMSGLEN // data 8 + ACE_CDR::MAX_ALIGNMENT; // padding; 9 10 ACE_OutputCDR payload (max_payload_size); 11 payload << log_record; 12 ACE_CDR::ULong length = payload.total_length (); 13 First marshal the payload to contain the linearized ACE_Log_Record
89

4. Then marshal the header info that includes byte order & payload length
14 15 16 17 18 19 20 21 22 23 24 25 } ACE_OutputCDR header (ACE_CDR::MAX_ALIGNMENT + 8); header << ACE_OutputCDR::from_boolean (ACE_CDR_BYTE_ORDER); header << ACE_CDR::ULong (length);
5. Construct an iovec of size 2 with header & payload info

iovec iov[2]; iov[0].iov_base iov[0].iov_len iov[1].iov_base iov[1].iov_len = = = = header.begin ()->rd_ptr (); 8; payload.begin ()->rd_ptr (); length;
return logging_peer_.sendv_n (iov, 2);
6. Send entire message to logging server

90
Since TCP/IP is a bytestream protocol (i.e., without any message boundaries) the logging service uses CDR as a message framing protocol to delimit log records

1 int main (int argc, char *argv[]) The Logging_Client 2 { 3 u_short logger_port = main program 4 argc > 1 ? atoi (argv[1]) : 0; 5 const char *logger_host = 6 argc > 2 ? argv[2] : ACE_DEFAULT_SERVER_HOST; 7 int result; 8 9 ACE_INET_Addr server_addr; 10 11 if (logger_port != 0) 12 result = server_addr.set (logger_port, logger_host); 13 else 14 result = server_addr.set ("ace_logger", logger_host); 15 if (result == -1) 16 ACE_ERROR_RETURN((LM_ERROR, 17 "lookup %s, %p\n", 18 logger_port == 0 ? "ace_logger" : argv[1], 19 logger_host), 1); 20
91
Sidebar: ACE Debugging & Error Macros

Consolidates printing of debug and error messages via a printf ()-like format e.g., ACE_DEBUG, ACE_ERROR (& their *_RETURN counterparts) that encapsulate the ACE_Log_Msg::log() method Arguments are enclosed in a double set of parentheses to make it appear as one argument to the C++ preprocessor First argument is the severity code; second one is a format string supporting a superset of printf() conversion specifiers Format
%l %N %n %P %p %T %t
92
Action
Displays the line number where the error occurred Displays the file name where the error occurred Displays the name of the program Displays the current process ID Takes a const char * argument and displays it and the error string corresponding to errno (similar to perror()) Displays the current time Displays the calling threads ID

Use the ACE_SOCK_Connector wrapper faade to connect to the logging server 21 22 23 24 25 26 27 28 29 30 31 32 ACE_SOCK_Connector connector; Logging_Client logging_client; if (connector.connect (logging_client.peer (), server_addr) < 0) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "connect()"), 1); // Limit the number of characters read on each record. cin.width (ACE_Log_Record::MAXLOGMSGLEN);
Contents of the message to be sent to logging server are obtained from standard input
93

33 for (;;) { 34 std::string user_input; 35 getline (cin, user_input, '\n'); 36 Create log_record 37 if (!cin || cin.eof ()) break; 38 39 ACE_Time_Value now (ACE_OS::gettimeofday ()); 40 ACE_Log_Record log_record (LM_INFO, now, 41 ACE_OS::getpid ()); 42 log_record.msg_data (user_input.c_str ()); 43 44 if (logging_client.send (log_record) == -1) 45 ACE_ERROR_RETURN ((LM_ERROR, 46 "%p\n", "logging_client.send()"), 1); 47 } Send log_record to logging server 48 49 return 0; // Logging_Client destructor 50 // closes TCP connection. 51 }
94
The Logging_Server Classes

The figure below illustrates our Logging_Server abstract base class, the Logging_Handler class we'll describe shortly, & the concrete logging server classes that we'll develop in subsequent sections of the tutorial
95
Implementing the Logging_Server (1/5)

This example uses the ACE_Message_Block & ACE CDR classes in a common base class that simplifies logging server implementations in the examples
// Forward declaration. class ACE_SOCK_Stream; Header file Logging_Server.h class Logging_Server { public: // Template Method that runs logging server's event loop. virtual int run (int argc, char *argv[]);
protected: // The following four methods are ``hooks'' that can be // overridden by subclasses. virtual int open (u_short logger_port = 0); virtual int wait_for_multiple_events () { return 0; } virtual int handle_connections () = 0; virtual int handle_data (ACE_SOCK_Stream * = 0) = 0;
96
Sidebar: Template Method Pattern

Intent Abstract Class Define the skeleton of an template_method (); algorithm in an operation, hook_method_1(); deferring some steps to hook_method_1(); ... subclasses hook_method_1(); ... Context hook_method_2(); ... You have a fixed algorithm structure with variations possible for individual steps Concrete Class 1 Problem You want to plug in & out steps hook_method_1(); of the algorithm without hook_method_2(); changing the algorithm itself Solution Concrete Class 2 Define a fixed base class function that calls virtual hook hook_method_2(); methods that derived classes can override 97

Header file Logging_Server.h (contd) // This helper method can be used by the hook methods. int make_log_file (ACE_FILE_IO &, ACE_SOCK_Stream * = 0);
// Close the socket endpoint and shutdown ACE. virtual ~Logging_Server () { acceptor_.close (); } // Accessor. ACE_SOCK_Acceptor &acceptor () { return acceptor_; }
private: // Socket acceptor endpoint. ACE_SOCK_Acceptor acceptor_; };
98

Implementation file Logging_Server.cpp
#include #include #include #include #include #include "ace/FILE_Addr.h" Template method providing the "ace/FILE_Connector.h" skeleton of the algorithm to use "ace/FILE_IO.h" Hook methods will be overridden by "ace/INET_Addr.h" subclasses unless default is ok to "ace/SOCK_Stream.h" use "Logging_Server.h"
int Logging_Server::run (int argc, char *argv[]) { if (open (argc > 1 ? atoi (argv[1]) : 0) == -1) return -1; Three hook methods that can be overridden in subclasses for (;;) { if (wait_for_multiple_events () == -1) return -1; if (handle_connections () == -1) return -1; if (handle_data () == -1) return -1; } return 0; }
99

Initialize the acceptor so it can accept connections from any server network interface
int Logging_Server::open (u_short logger_port) { // Raises the number of available socket handles to // the maximum supported by the OS platform. ACE::set_handle_limit (); ACE_INET_Addr server_addr; int result; if (logger_port != 0) result = server_addr.set (logger_port, INADDR_ANY); else result = server_addr.set ("ace_logger", INADDR_ANY); if (result == -1) return -1;
// Start listening and enable reuse of listen address // for quick restarts. return acceptor_.open (server_addr, 1);
100

int Logging_Server::make_log_file (ACE_FILE_IO &logging_file, ACE_SOCK_Stream *logging_peer) { std::string filename (MAXHOSTNAMELEN, \0); if (logging_peer != 0) { // Use client host name as file name. ACE_INET_Addr logging_peer_addr; logging_peer->get_remote_addr (logging_peer_addr); logging_peer_addr.get_host_name (filename.c_str (), filename.size ()); filename += ".log"; } else filename = "logging_server.log";
ACE_FILE_Connector connector; return connector.connect (logging_file, ACE_FILE_Addr (filename.c_str ()), 0, // No time-out. ACE_Addr::sap_any, // Ignored. Create the log file using the 0, // Don't try to reuse the addr. ACE_FILE_Connector factory O_RDWR|O_CREAT|O_APPEND, ACE_DEFAULT_FILE_PERMS); 101 }
Sidebar: The ACE File Wrapper Facades

ACE file wrapper facades encapsulate platform mechanisms for unbuffered file operations
The design of these wrapper facades is very similar to ACE IPC wrapper facades
The ACE File classes decouple: Initialization factories: e.g., ACE_FILE_Connector, which opens and/or creates files Data transfer classes: e.g., ACE_FILE_IO, which applications use to read/write data from/to files opened using ACE_FILE_Connector This generality in ACEs design of wrapper facades helps strategize higher-level ACE framework components e.g., ACE_Acceptor, ACE_Connector, & ACE_Svc_Handler
102
Implementing the Logging_Handler (1/6)

Header file Logging_Handler.h
#include "ace/FILE_IO.h" #include "ace/SOCK_Stream.h" class ACE_Message_Block; // Forward declaration. class Logging_Handler { protected: // Reference to a log file. ACE_FILE_IO &log_file_; // Connected to the client. ACE_SOCK_Stream logging_peer_; This class is used by the logging server to encapsulate the I/O & processing of log records
103

Header file Logging_Handler.h contd
// Receive one log record from a connected client. <mblk> // contains the hostname, <mblk->cont()> contains the log // record header (the byte order and the length) and the data. int recv_log_record (ACE_Message_Block *&mblk); // Write one record to the log file. The <mblk> contains the // hostname and the <mblk->cont> contains the log record. int write_log_record (ACE_Message_Block *mblk);
// Log one record by calling <recv_log_record> and // <write_log_record>. int log_record (); };
When a log record is received it is stored as an ACE_Message_Block chain

104

1. Receive incoming data & use the input CDR class to parse header 2. Then payload based on the framing protocol & 3. Finally save it in an ACE_Message_Block chain
1 int Logging_Handler::recv_log_record (ACE_Message_Block *&mblk) 2 { First save the peer hostname 3 ACE_INET_Addr peer_addr; 4 logging_peer_.get_remote_addr (peer_addr); 5 mblk = new ACE_Message_Block (MAXHOSTNAMELEN + 1); 6 peer_addr.get_host_name (mblk->wr_ptr (), MAXHOSTNAMELEN); 7 mblk->wr_ptr (strlen (mblk->wr_ptr ()) + 1); // Go past name. 8 9 ACE_Message_Block *payload = 10 new ACE_Message_Block (ACE_DEFAULT_CDR_BUFSIZE); 11 // Align Message Block for a CDR stream. Force proper alignment 12 ACE_CDR::mb_align (payload); 13 14 if (logging_peer_.recv_n (payload->wr_ptr (), 8) == 8) { 15 payload->wr_ptr (8); // Reflect addition of 8 bytes. 16 Receive the header info (byte 17 ACE_InputCDR cdr (payload); order & length) 18 105

ACE_CDR::Boolean byte_order; Demarshal header info // Use helper method to disambiguate booleans from chars. cdr >> ACE_InputCDR::to_boolean (byte_order); cdr.reset_byte_order (byte_order); ACE_CDR::ULong length; Resize message block to be the right size for payload & thats aligned properly cdr >> length; payload->size (8 + ACE_CDR::MAX_ALIGNMENT + length); if (logging_peer_.recv_n (payload->wr_ptr(), length) > 0) { payload->wr_ptr (length); // Reflect additional bytes. mblk->cont (payload); // Chain the header and payload. return length; // Return length of the log record. } } payload->release (); mblk->release (); payload = mblk = 0; return -1; On error, free up allocated message blocks
19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 }
106

1. Send the message block chain to the log file, which is stored in binary format 2. If debug flag is set, print contents of the message
1 int Logging_Handler::write_log_record (ACE_Message_Block *mblk) 2 { 3 if (log_file_->send_n (mblk) == -1) return -1; 4 5 if (ACE::debug ()) { 6 ACE_InputCDR cdr (mblk->cont ()); 7 ACE_CDR::Boolean byte_order; 8 ACE_CDR::ULong length; 9 cdr >> ACE_InputCDR::to_boolean (byte_order); 10 cdr.reset_byte_order (byte_order); 11 cdr >> length; 12 ACE_Log_Record log_record; 13 cdr >> log_record; // Extract the <ACE_log_record>. 14 log_record.print (mblk->rd_ptr (), 1, cerr); 15 } 16 17 return mblk->total_length (); 18 } 107

1. Receives the message 2. Demarshals it into a ACE_Message_Block & 3. Writes it to the log file int Logging_Handler::log_record () { ACE_Message_Block *mblk = 0; if (recv_log_record (mblk) == -1) return -1; else { int result = write_log_record (mblk); mblk->release (); // Free up the entire contents. return result == -1 ? -1 : 0; } }
Later on well see the virtue of splitting the recv_log_record() & write_log_record() logic into two methods
108
Iterative Logging Server

This is the simplest possible logging server implementation The iterative server implementation simply accepts & processes one client connection at a time Clearly, this approach does not scale up for non-trivial applications of the logging service!!! Subsequent implementations will enhance this version, yet still use the logging server framework
109
ACE_SOCK_Acceptor ACE_SOCK_Stream
Only one client is accepted/processed at a time
Implementing the Iterative_Logging_Server (1/3)

#include #include #include #include #include "ace/FILE_IO.h" "ace/INET_Addr.h" "ace/Log_Msg.h" "Logging_Server.h" "Logging_Handler.h"
Header file: Iterative_Logging_Server.h
class Iterative_Logging_Server : public Logging_Server { public: Iterative_Logging_Server (): logging_handler_ (log_file_) {}
Logging_Handler &logging_handler () { return logging_handler_; }

protected: ACE_FILE_IO log_file_; Logging_Handler logging_handler_; // Other methods shown below... };
110

virtual int open (u_short logger_port) { if (make_log_file (log_file_) == -1) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "make_log_file()"), -1); return Logging_Server::open (logger_port); }
Override & decorate the Logging_Server::open() method
virtual int handle_connections () { ACE_INET_Addr logging_peer_addr;
Override the handle_connections() hook method to handle one connection at a time
if (acceptor ().accept (logging_handler_.peer (), &logging_peer_addr) == -1) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "accept()"), -1); ACE_DEBUG ((LM_DEBUG, "Accepted connection from %s\n", logging_peer_addr.get_host_name ())); return 0; }
111

virtual int handle_data (ACE_SOCK_Stream *) { while (logging_handler_.log_record () != -1) continue;
Delegate I/O to Logging_Handler
logging_handler_.close (); // Close the socket handle. return 0; }
Main program of iterative logging server

#include "ace/Log_Msg.h" #include "Iterative_Logging_Server.h" int main (int argc, char *argv[]) { Iterative_Logging_Server server; if (server.run (argc, argv) == -1) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "server.run()"), 1); return 0; }
112
Concurrency Design Dimensions

Concurrency is essential to develop scalable & robust networked applications, particularly servers The next group of slides present a domain analysis of concurrency design dimensions that address the policies & mechanisms governing the proper use of processes, threads, & synchronizers We cover the following design dimensions in this chapter: Iterative versus concurrent versus reactive servers Processes versus threads Process/thread spawning strategies User versus kernel versus hybrid threading models Time-shared versus real-time scheduling classes Task- versus message-based architectures
113
Iterative vs. Concurrent Servers
Iterative/reactive servers handle each client request in its entirety before servicing subsequent requests Best suited for short-duration or 114 infrequent services
Concurrent servers handle multiple requests from clients simultaneously Best suited for I/O-bound services or long-duration services
Also good for busy servers
Multiprocessing vs. Multithreading
A process provides the context for executing program instructions Each process manages certain resources (such as virtual memory, I/O handles, and signal handlers) & is protected from other OS processes via an MMU IPC between processes can be complicated & inefficient
115
A thread is a sequence of instructions in the context of a process Each thread manages certain resources (such as runtime stack, registers, signal masks, priorities, & thread-specific data) Threads are not protected from other threads IPC between threads can be more efficient than IPC between processes
Thread Pool Eager Spawning Strategies

This strategy prespawns one or more OS processes or threads at server creation time These``warm-started'' execution resources form a pool that improves response time by incurring service startup overhead before requests are serviced Two general types of eager spawning strategies are shown below:
These strategies based on Half-Sync/Half-Async & Leader/Followers patterns

116
The Half-Sync/Half-Async Pattern

The Half-Sync/Half-Async pattern decouples async & sync service processing in concurrent systems to simplify programming without unduly reducing performance
Sync Service Layer Sync Service 1 Sync Service 2 Sync Service 3
<<read/write>> Queueing Layer
<<read/write>> Queue <<read/write>>
Async Service Layer
<<dequeue/enqueue>> Async Service
<<interrupt>> External Event Source
This pattern yields two primary benefits: 1.Threads can be mapped to separate CPUs to scale up server performance via multi-processing 2.Each thread blocks independently, which prevents a flowcontrolled connection from degrading the QoS that other clients receive
117
Half-Sync/Half-Async Pattern Dynamics

: External Event Source : Async Service : Queue : Sync Service
notification read() work() message
message enqueue() notification read() work() message
This pattern defines two service processing layersone async & one syncalong with a queueing layer that allows services to exchange messages between the two layers
118
The pattern allows sync services (such as processing log records from different clients) to run concurrently, relative both to each other & to async/reactive services (such as event demultiplexing)
Drawbacks with Half-Sync/Half-Async

Problem
Although Half-Sync/Half-Async threading model is more scalable than the purely reactive model, it is not necessarily the most efficient design e.g., passing a request between the async thread & a worker thread incurs: Dynamic memory (de)allocation, Synchronization operations, A context switch, & CPU cache updates This overhead can make server latency unnecessarily high
Worker Thread 1
<<get>>
Worker Thread 2
<<get>>
Worker Thread 3
<<get>>
Request Queue
<<put>>
handlers acceptor
Event source
Solution
Apply the Leader/Followers architectural pattern to minimize server threading overhead
119
The Leader/Followers Pattern

The Leader/Followers architectural pattern is more efficient than HalfSync/Half-Async Multiple threads take turns sharing event sources to detect, demux, dispatch, & process service requests that occur on the event sources This pattern eliminates the need for& the overhead ofa separate Reactor thread & synchronized request queue used in the Half-Sync/Half-Async pattern
Handles
Concurrent Handles Handle Sets Concurrent Handle Sets Iterative Handle Sets 120 UDP Sockets +
WaitForMultipleObjects()
demultiplexes
Thread Pool synchronizer join() promote_new_leader()

uses
*
Event Handler handle_event () get_handle()
Handle
*
Handle Set handle_events() deactivate_handle() reactivate_handle() select() Concrete Event Handler B handle_event () get_handle() Concrete Event Handler A handle_event () get_handle()
Iterative Handles TCP Sockets +

WaitForMultpleObjects()
UDP Sockets + select()/poll()
TCP Sockets + select()/poll()
Leader/Followers Pattern Dynamics

Thread 1 Thread 2 : Thread Pool : Handle Set : Concrete Event Handler
1.Leader thread demuxing 2.Follower thread promotion 3.Event handler demuxing & event processing 4.Rejoining the thread pool
121
join() handle_events() join() event handle_event() deactivate_ handle() promote_ new_leader()
thread 2 sleeps until it becomes the leader thread 2 waits for a new event, thread 1 processes current event join() thread 1 sleeps until it becomes the leader
handle_ events()
reactivate_ handle()
event
handle_event() deactivate_ handle()
Thread-per-Request On-demand Spawning Strategy

On-demand spawning creates a new process or thread in response to the arrival of client connection and/or data requests Typically used to implement the thread-per-request and thread-perconnection models
The primary benefit of on-demand spawning strategies is their reduced consumption of resources The drawbacks, however, are that these strategies can degrade performance in heavily loaded servers & determinism in real-time systems due to costs of spawning processes/threads and starting services
122
The N:1 & 1:1 Threading Models

OS scheduling ensures applications use host CPU resources suitably Modern OS platforms provide various models for scheduling threads A key difference between the models is the contention scope in which threads compete for system resources, particularly CPU time The two different contention scopes are shown below:
123
Process contention scope (aka user threading) where threads in the same process compete with each other (but not directly with threads in other processes)
System contention scope (aka kernel threading) where threads compete directly with other system-scope threads, regardless of what process theyre in
The N:M Threading Model

Some operating systems (such as Solaris) offer a combination of the N:1 & 1:1 models, referred to as the ``N:M' hybridthreading model When an application spawns a thread, it can indicate in which contention scope the thread should operate The OS threading library creates a user-space thread, but only creates a kernel thread if needed or if the application explicitly requests the system contention scope
124
When the OS kernel blocks an LWP, all user threads scheduled onto it by the threads library also block However, threads scheduled onto other LWPs in the process can continue to make progress
Task- vs. Message-based Concurrency Architectures

A concurrency architecture is a binding between: CPUs, which provide the execution context for application code Data & control messages, which are sent & received from one or more applications & network devices Service processing tasks, which perform services upon messages as they arrive & depart
Task-based concurrency architectures structure multiple CPUs according to units of service functionality in an application Message-based concurrency architectures structure the CPUs according to the messages received from applications & network devices 125
Overview of OS Concurrency Mechanisms

Networked applications, particularly servers, must often process requests concurrently to meet their quality of service requirements This section presents an overview of the Synchronous event demultiplexing
Multiprocessing
Multithreading & Synchronization mechanisms available to implement those designs We also discuss common portability & programming problems that arise when networked applications are developed using native C-level concurrency APIs
126
Synchronous Event Demultiplexing

Synchronous event demuxers wait for certain events to occur on a set of event sources, where the caller is returned the thread of control whenever one or more event sources become active e.g., poll() on System V UNIX, WaitForMultipleObjects() on Win32, & select() select() is the most common event demultiplexing API for I/O handles int select (int width, // Maximum handle plus 1 fd_set *read_fds, // Set of "read" handles fd_set *write_fds, // Set of "write" handles fd_set *except_fds, // Set of "exception" handles struct timeval *timeout);// Time to wait for events fd_set is a structure select() modifies the fd_set depending on the representing the set active/inactive handles as follows: of handles to check If a handle is not active in the fd_set, it is ignored & select for I/O events, such will keep it inactive in the fd_set as ready for reading, If a handle is active, select() will determine if there are writing, or exception pending events. If there is one, the appropriate fd_set has events the handle activated else its value is made inactive in the returned fd_set
127
Multiprocessing Mechanisms
Multiprocessing mechanisms include the features provided by the OS for creating & managing the execution of multiple processes, e.g., Process lifetime operations such as fork() & exec*() on POSIX & CreateProcess() on Win32 create a new process address space for programs to run The initiating process can set command line arguments, environment variables and working directories for the new process The new process can terminate voluntarily by reaching the end of its execution or be involuntarily killed via signals (in POSIX) or TerminateProcess() (in Win32) Process synchronization options provided by the OS to retain the identity and exit status of a process and report it to the parent, e.g., POSIX wait() & waitpid() Win32 WaitForSingleObject() & WaitForMultipleObjects() Process property operations used to get/set process properties, such as default file access permissions, user identification, resource limits, scheduling priority and current working directory
128
Multithreading Mechanisms (1/2)

Multithreading mechanisms are provided by the OS to handle thread lifetime management, synchronization, priorities, & thread specific storage Thread lifetime operations include operations to create threads, e.g., pthread_create() (PThreads) & CreateThread() (Win32) Thread termination is achieved in the following manner: Voluntarily by reaching the end point of the thread entry function or calling pthread_exit() (Pthreads) or ExitThread() (Win32)
Involuntarily by being killed via a signal or an aynchronous thread cancelation operations, such as pthread_cancel() (Pthreads) and TerminateThread() (Win32)
Thread synchronization operations that allow created threads to be Detached where the OS reclaims storage used for the threads state & exit status after it has exit Joinable where the OS retains identity & exit status of a terminating thread so other threads can synchronize with it Other operations that allow threads to suspend & resume each other, or send signals to other threads
129
Multithreading Mechanisms (2/2)

Thread property operations includes operations to set and get thread properties, such as priority and scheduling class.
Thread-specific storage is similar to global data except that the data is global in the scope of the executing thread.
Each thread has its own copy of a TSS data e.g., errno Each TSS item has a key that is global to all threads within a process A thread uses this key to access its copy of the TSS data Keys are created by factory functions, such as pthread_key_create() (Pthreads) or TlsAlloc() (Win32). Key/pointer relationships are managed by TSS set/get functions, such as pthread_getspecific() & pthread_setspecific() (Pthreads) or TlsGetValue() & TlsSetValue() (Win32)
130
Synchronization Mechanisms (1/2)

Synchronization mechanisms allow processes and threads to coordinate their execution order and the order in which they access shared resources, such as files, network devices, database records, and shared memory Mutexes serialize execution of multiple threads by defining a critical section of code that can be executed by only one thread at a time. A thread owning a mutex must release it
There are two kinds of mutexes:

Nonrecursive mutex that will deadlock or fail if the thread currently owning the mutex tries to reacquire it without first releasing it Recursive mutex that will allow the thread owning the mutex to reacquire it without deadlocking The owner thread is responsible to release it the same number of times it has acquired it
131
Synchronization Mechanisms (2/2)

Readers/writer locks allows access to a shared resource by either multiple threads simultaneously having read-only access or only one thread at a time having a read-write access They help improve performance for applications where resources are read more often than modified They can be implemented to give more preference to either the readers or the writer
Semaphores is a non negative integer that can be incremented and decremented atomically
A thread blocks when it tried to decrement a semaphore whose value is 0 A block thread makes progress when another thread increments the value of the semaphore Usually implemented using sleep locks, that trigger a context switch Condition variables allows a thread to coordinate & schedule its own processing A thread can wait on complex expressions to attain a desired state Used to build higher level patterns such as active object & monitor objects
132
Sidebar: Evaluating Synchronization Mechanisms

Performance of synchronization mechanisms depends on the OS implementation and hardware
Some general issues to keep in mind: Condition variables & semaphores generally have a higher overhead than mutexes Native OS implementations usually perform better than emulated behavior Mutexes versus Reader/Writer Locks mutexes generally have the lower overhead than reader/writer locks. On a multiprocessor platform, reader/writer locks scale well since multiple readers can execute in parallel Nonrecursive mutexes are more efficient than recursive mutexes Moreover, subtle errors can be caused using recursive mutexes due to mismatch in the number of lock & unlock operations
133
Sidebar: ACE API Error Propagation Strategies

Error reporting strategies usually differ across different concurrency APIs & OS e.g., UI & Pthreads return 0 on success and a non-zero number on failure whereas Win32 returns 0 on failure and conveys the error value via thread specific storage
This makes code non-portable and filled with accidental complexities

ACE concurrency wrapper facades solve this problem by returning -1 on error and setting the errno variable in thread specific storage
134
The ACE Event Demuxing Wrapper Facades

The reactive server model can be thought of as lightweight multitasking, where a single-threaded server communicates with multiple clients in a round-robin manner without introducing the threading & synchronization overhead & complexity This server concurrency strategy uses an event loop that examines & reacts to events from its clients continuously An event loop demultiplexes input from various event sources so they can be processed in an orderly way Event sources in networked applications are primarily socket handles The most popular event demultipelxing function is select(), which provides the basis for the ACE classes described below
135
The ACE_Handle_Set Class (1/2)

Motivation The fd_set represents a source of accidental complexity in the following areas: The code to scan for active handles is often a hot spot since it executes continually in a tight loop The macros supplied to manipulate & scan an fd_set must be used carefully to avoid processing handles that aren't active & to avoid corrupting an fd_set The fd_set is defined in system-supplied header files whose representation is exposed to programmers There are subtle nonportable aspects of fd_set when used in conjunction with select()
136
The ACE_Handle_Set Class (2/2)

Class Capabilities The ACE_Handle_Set class uses the Wrapper Faade pattern to encapsulate fd_sets & provide the following capabilities: It enhances portability, ease of use, & type safety of event-driven applications that use of fd_set & select() across OS platforms It tracks & adjusts the fd_set size-related values automatically as handles are added & removed ACE_Handle_Set_Iterator is an optimized iterator for ACE_Handle_Set
137
Reactive Logging Server Version 1

This example enhances the earlier iterative logging server implementation by using select() together with the ACE_Handle_Set & ACE_Handle_Set_Iterator classes This server demultiplexes the following two types of events: Arrival of new connections from clients Arrival of log records on client connections
138
Using the ACE_Handle_Set (1/4)

We use a pair of ACE_Handle_Set data members since select() modifies the fd_set parameters passed to it, so we need to keep a master copy. class Reactive_Logging_Server : public Iterative_Logging_Server { protected: // Keeps track of the acceptor socket handle and all the // connected stream socket handles. ACE_Handle_Set master_handle_set_; // Keep track of handles marked as active by <select>. ACE_Handle_Set active_handles_; typedef Iterative_Logging_Server PARENT; // Other methods shown below... };
139

virtual int open (u_short logger_port) { PARENT::open (logger_port); master_handle_set_.set_bit (acceptor ().get_handle ()); acceptor ().enable (ACE_NONBLOCK); return 0; open() method uses ACE_Handle_Set to track the } acceptors handle in fd_set Note the use of non-blocking acceptor... virtual int wait_for_multiple_events () { active_handles_ = master_handle_set_;
Override hook method
140
if (select (active_handles_.max_set () + 1, active_handles_.fdset (), 0, // no write_fds 0, // no except_fds Use select() to determine 0) == -1) // no timeout all active handles return -1; active_handles_.sync ((ACE_HANDLE) active_handles_.max_set () + 1); return 0; sync() resets handle count in ACE_Handle_Set after the select() call }
Sidebar: Motivation for Non-blocking Acceptors

Context An acceptor socket is passed to select() & gets marked as active when a connection is received Many servers use this event to call accept() without blocking Problem Client disconnects exactly within the time interval between the server making the select() & accept() calls
Possible race condition due to asynchronous behavior of TCP/IP leading to accept() call being blocked forever & hanging the application
Solution Acceptor sockets should always be set in non-blocking mode Achieved portably in ACE via the enable()method of ACE_IPC_SAP class passing it the ACE_NONBLOCK flag
141

Override the hook method
virtual int handle_connections () { if (active_handles_.is_set (acceptor ().get_handle ())) { while (acceptor ().accept (logging_handler ().peer ()) == 0) master_handle_set_.set_bit (logging_handler ().peer ().get_handle ());
// Remove acceptor handle from further consideration. active_handles_.clr_bit (acceptor ().get_handle ());
} return 0; }
If the acceptor() handle is active, iteratively accept all the connections & save them in master_handle_set_
142

Reactive logging server main program
#include "ace/Log_Msg.h" #include "Reactive_Logging_Server.h" int main (int argc, char *argv[]) { Reactive_Logging_Server server; if (server.run (argc, argv) == -1) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "server.run()"), 1); return 0; }
handle_data() method shown later using ACE_Handle_Set_Iterator class

143
Using the ACE_Handle_Set_Iterator (1/2)

This method has problems. Do not copy this example!
virtual int handle_data (ACE_SOCK_Stream *) { for (ACE_HANDLE handle = acceptor ().get_handle () + 1; handle < active_handles_.max_set () + 1; handle++) { Non-portable assumption that socket handles are a contiguous range integers Cannot assume acceptor socket has lowest number if (active_handles_.is_set (handle)) { logging_handler ().peer ().set_handle (handle); if (logging_handler ().log_record () == -1) { // Handle connection shutdown or comm failure. master_handle_set_.clr_bit (handle); logging_handler ().close (); } } Its inefficient to sequentially search } for active handles in the large, but return 0; sparsely populated handle set
144
Using the ACE_Handle_Set_Iterator (2/2)

Overcoming drawbacks outlined before in handle_data() method by using the ACE_Handle_Set_Iterator class virtual int handle_data (ACE_SOCK_Stream *) { ACE_Handle_Set_Iterator peer_iterator (active_handles_); for (ACE_HANDLE handle; (handle = peer_iterator ()) != ACE_INVALID_HANDLE; ) { logging_handler ().peer ().set_handle (handle); if (logging_handler ().log_record () == -1) { // Handle connection shutdown or comm failure. master_handle_set_.clr_bit (handle); logging_handler ().close (); } } } ACE_Handle_Set_Iterator optimizes searching for active handles based on the underlying platforms representation of fd_set These optimizations are encapsulated within a common interface, making the 145 class portable
Reactive Logging Server Version 2

We now extend our reactive server example to write log records from different clients to different log files, one for each connected client This reactive server implementation maintains a map container that allows a logging server to keep separate log files for each of its clients
The figure also shows how we use the ACE::select() wrapper method & the ACE_Handle_Set class to service multiple clients via a reactive server model
146
Implementing the Reactive Logging Server (1/6)

class ACE { public: static int select (int width, ACE_Handle_Set &rfds, const ACE_Time_Value *tv = 0); static int select (int width, ACE_Handle_Set *rfds, ACE_Handle_Set *wfds = 0, ACE_Handle_Set *efds = 0, const ACE_Time_Value *tv = 0); // ... Other methods omitted ... }; #include #include #include #include #include #include "ace/ACE.h" Reactive logging server using "ace/Handle_Set.h" "ace/Hash_Map_Manager.h" ACE::select() & hash map container classes "ace/Synch.h" "Logging_Server.h" "Logging_Handler.h"
typedef ACE_Hash_Map_Manager<ACE_HANDLE, ACE_FILE_IO *, ACE_Null_Mutex> LOG_MAP;

147

Association between each connected peer & its log file is maintained in a hash map
class Reactive_Logging_Server_Ex : public Logging_Server { protected: // Associate an active handle to an <ACE_FILE_IO> pointer. LOG_MAP log_map_;
// Keep track of acceptor socket and all the connected // stream socket handles. ACE_Handle_Set master_handle_set_; // Keep track of read handles marked as active by <select>. ACE_Handle_Set active_read_handles_; typedef Logging_Server PARENT; // Other methods shown below... };
148

open() method similar to previous one for first version of reactive logging server
virtual int open (u_short logger_port) { PARENT::open (logger_port); master_handle_set_.set_bit (acceptor ().get_handle ()); acceptor ().enable (ACE_NONBLOCK); return 0; } virtual int wait_for_multiple_events () { active_read_handles_ = master_handle_set_; int width = (int) active_read_handles_.max_set () + 1; return ACE::select (width, active_read_handles_); } Note use of ACE::select(), which calls active_read_handles_.sync() automatically
149

This version of reactive logging server creates a log file when a peer establishes connection The association between the log file & the handle is maintained in the hash map virtual int handle_connections () { ACE_SOCK_Stream logging_peer; while (acceptor ().accept (logging_peer) != -1) { ACE_FILE_IO *log_file = new ACE_FILE_IO; // Use the client's hostname as the logfile name. make_log_file (*log_file, &logging_peer); // Add the new <logging_peer>'s handle to the map and // to the set of handles we <select> for input. log_map_.bind (logging_peer.get_handle (), log_file); master_handle_set_.set_bit (logging_peer.get_handle ()); } // Remove acceptor handle from further consideration... active_read_handles_.clr_bit (acceptor ().get_handle ()); return 0;
150

virtual int handle_data (ACE_SOCK_Stream *) { ACE_Handle_Set_Iterator peer_iterator (active_read_handles_); for (ACE_HANDLE handle; (handle = peer_iterator ()) != ACE_INVALID_HANDLE; ) { Identify the log file corresponding to the peer ACE_FILE_IO *log_file; who sent the data log_map_.find (handle, log_file); Logging_Handler logging_handler (*log_file, handle); if (logging_handler.log_record () == -1) { logging_handler.close (); master_handle_set_.clr_bit (handle); log_map_.unbind (handle); log_file->close (); delete log_file; Free up resources on error } } return 0;
151

Reactive logging server main program (this should start looking rather familiar by now ;-))
int main (int argc, char *argv[]) { Reactive_Logging_Server_Ex server;
if (server.run (argc, argv) == -1) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "server.run()"), 1); return 0; }
152
The ACE Process Wrapper Facades

OS multiprocessing support helps to Enable concurrency by scheduling & running separate processes on different CPUs Increase robustness by using memory-management unit (MMU) hardware to protect separate process address spaces from accidental or malicious corruption by other active processes in the system Enhance security by allowing each process to verify or control per-user or persession security & authentication information We now cover the following ACE classes that can use to spawn & manage one or more processes:
These classes are related as follows:
153
The ACE_Process Class (1/2)

Motivation OS process management mechanisms differ syntactically & semantically UNIX
e.g., the UNIX fork() system function is very different than Windows CreateProcess() Addressing these platform variations in each application is difficult, tedious, error prone, & unnecessary since ACE provides the ACE_Process class
154
Windows
The ACE_Process Class (2/2)

Class Capabilities This class encapsulates the variation among different OS multiprocessing APIs in accordance with the Wrapper Facade pattern by defining methods that provide the following capabilities: Spawn & terminate processes Synchronize on process exit Access process properties, such as process ID
Most methods are portable, though some are platform-specific

155
Sidebar: POSIX Portability Challenges

Context Many Unix applications use POSIX fork() system call to build concurrent process-based servers
Problem Portability is a concern if we rely on fork() to both:

Have a parent process fork() a child process with a duplicate of the parents address space and all I/O handles Have both parent and child return from fork() with the same location but with different return values, which can then run different parts of the program Forces fork() is the only system call that allows duplicating an address space I/O handles of the parent are often required in the child process Solution Use the ACE_Process_Options class that provides methods such as pass_handle(), working_directory(), setenv() & others
156
Using ACE_Process
This program uses ACE_Process to computes factorials recursively
int main (int argc, char *argv[]) { ACE_Process_Options options; char *n_env = 0; int n; if (argc == 1) { // Top-level process. n_env = ACE_OS::getenv ("FACTORIAL"); n = n_env == 0 ? 0 : atoi (n_env); options.command_line ("%s %d", argv[0], n == 0 ? 10 : n); } else if (atoi (argv[1]) == 1) return 1; // Base case. else { n = atoi (argv[1]); options.command_line ("%s %d", argv[0], n - 1); } Pass command line for spawned child to use ACE_Process child; child.spawn (options); // Make the ``recursive'' call. child.wait (); return n * child.exit_code (); // Compute n factorial. }
157
The ACE_Process_Options Class (2/2)

Class Capabilities This class unifies how process properties are passed to ACE_Process & ACE_Process_Manager to provide the following capabilities: Enable an application to specify desired process control information Allow process control items to be expanded as platforms change Provide a decoupling mechanism that enables ACE to offer these capabilities without varying the process creation interface
158
The ACE_Process_Options Class (1/2)

Motivation Operating systems provide various methods for setting the properties of newly-created processes, including Program image, which program should the new process execute? Open I/O handles, e.g., should the child process inherit open I/O handles or other OS objects; should it close some or all of its inherited open handles? Working directory, e.g., should the child process run in the same or different directory as its parent? Process relationship, e.g., should the child process run in the background as an independent daemon or as part of a related group?
159
Using ACE_Process_Options (1/2)

This example demonstrates how to use ACE_Process_Options to pass environment, working directory, & command line to spawned child process
int main (int argc, char *argv[]) { ACE_Process_Options options; FILE *fp = 0; char *n_env = 0; int n; if (argc == 1) { // Top-level process. n_env = ACE_OS::getenv ("FACTORIAL"); n = n_env == 0 ? 0 : atoi (n_env); options.command_line ("%s %d", argv[0], n == 0 ? 10 : n); const char *working_dir = ACE_OS::getenv ("WORKING_DIR"); if (working_dir) options.working_directory (working_dir); fp = fopen ("factorial.log", "a"); options.setenv ("PROGRAM=%s", ACE::basename (argv[0])); }
160
Using ACE_Process_Options (2/2)

else { fp = fopen ("factorial.log", "a"); if (atoi (argv[1]) == 1) { fprintf (fp, "[%s|%d]: base case\n", ACE_OS::getenv ("PROGRAM"), ACE_OS::getpid ()); fclose (fp); return 1; // Base case. } else { n = atoi (argv[1]); options.command_line ("%s %d", argv[0], n - 1); } Wait for child to exit so we can get its exit status (which } will be a factorial value it has computed for n-1) ACE_Process child; child.spawn (options); // Make the ``recursive'' call. child.wait (); int factorial = n * child.exit_code (); // Compute n factorial. fprintf (fp, "[%s|%d]: %d! == %d\n", ACE_OS::getenv ("PROGRAM"), ACE_OS::getpid (), n, factorial); fclose (fp); return factorial; 161 }
The ACE_Process_Manager Class (1/2)

Motivation Complex networked applications often require groups of processes to coordinate to provide a particular service, e.g., A multistage workflow automation application may spawn multiple processes to work on different parts of a large problem One master process may wait for the entire group of worker processes to exit before proceeding with the next stage in the workflow
162
The ACE_Process_Manager Class (2/2)

Class Capabilities This class uses the Wrapper Facade pattern to combine the portability & power of ACE_Process with the ability to manage groups of processes as a unit & provide the following capabilities: It provides internal record keeping to manage & monitor groups of processes that are spawned by the ACE_Process class It allows one process to spawn a group of process & wait for them to exit before proceeding with its own processing
163
Multiprocessing Logging Server

This revision of the logging server uses a process-per connection concurrency model The master process spawns a new worker process for each accepted connection to the logging service port The master process then continues to accept new connections Each worker process handles all logging requests sent by a client across one connection; the process exits when this connection closes
164
Using ACE_Process_Manager (1/6)

class Process_Per_Connection_Logging_Server : public Logging_Server { public: Process-per-connection // Methods shown below logging server that uses the protected: ACE_Process_Manager std::string prog_name_; }; virtual int run (int argc, char *argv[]) { prog_name_ = argv[0]; // Ensure NUL-termination. prog_name_[prog_name_.size () - 1] = '\0'; if (argc == 3) return run_worker (argc, argv); // Only on Win32. else return run_master (argc, argv); }
165
Process Creation on POSIX & Windows

UNIX
Master/Worker Process Creation Sequence for POSIX Note use of child copy of parents process image
Master/Worker Process Creation Sequence for Windows Child doesnt automatically inherit the parent processs image, so we must take other steps e.g., pass the socket handle
166
Windows

int run_master (int argc, char *argv[]) { u_short logger_port = 0; if (argc == 2) logger_port = atoi (argv[1]); if (open (logger_port) == -1) return -1; for (;;) if (handle_connections () == -1) return -1;
return 0;
} Method only gets invoked for Win32
int run_worker (int argc, char *argv[]) { ACE_HANDLE socket_handle = ACE_static_cast (ACE_HANDLE, atoi (argv[2])); ACE_SOCK_Stream logging_peer (socket_handle); handle_data (&logging_peer); logging_peer.close (); return 0;
}
167
Sidebar: Portable Casting on C++ Compilers

Context C language cast operator used frequently for type casting Problem Can lead to subtle errors due to type violations caused by misuse Although C++ introduces several keywords to allow robust type casting, they are not supported across the spectrum of C++ compilers Solution Use ACE cast macros shown below
ACE Cast Macro

ACE_const_cast (TYPE, EXPR) ACE_static_cast (TYPE, EXPR)
C++ Cast Used if Available

const_cast<TYPE> (EXPR) static_cast<TYPE> (EXPR)
ACE_dynamic_cast (TYPE, EXPR)

ACE_reinterpret_cast (TYPE, EXPR)
dynamic_cast<TYPE> (EXPR)
reinterpret_cast<TYPE> (EXPR)
These methods are largely deprecated in ACE since new C++ compilers are better
168

1 virtual int handle_connections () { 2 ACE_SOCK_Stream logging_peer; 3 if (acceptor ().accept (logging_peer) == -1) 4 return -1; Create a Logger_Process that the process manager will manage 5 6 Logging_Process *logger = 7 new Logging_Process (prog_name_, logging_peer); 8 ACE_Process_Options options; Note use of Singleton pattern 9 pid_t pid = ACE_Process_Manager::instance ()->spawn 10 (logger, options); spawn() invokes hook methods on logger 11 if (pid == 0) { 12 acceptor().close (); to initialize options properly 13 handle_data (&logging_peer); 14 delete logger; On POSIX, this will be a child, so close the acceptor, handle the data, & release 15 ACE_OS::exit (0); resources 16 } On Win32, the child process created after the spawn call above will end up running the run_worker() method shown earlier instead of lines 11 thru 16
169
Singleton Pattern
Intent Ensure a class has only one instance & provide a global point of access to it Context Need to address initialization versus usage ordering Problem Want to ensure a single instance of a class, shared by all uses throughout a program
class Singleton { public: static Singleton *instance (){ if (instance_ == 0) { instance_ = new Singleton; } return instance_; } void method_1 (); // Other methods omitted private: // Private constructor Singleton (); // Initialized to 0 by linker. static Singleton *instance_; };
Solution Provide a global access method (static in C++) First use of the access method instantiates the class Constructors for instance are made private
170

17 logging_peer.close (); Parent does not need the peer 18 if (pid == -1) 19 ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "spawn()"), -1); 20 21 return ACE_Process_Manager::instance ()->wait 22 (0, ACE_Time_Value::zero); 23 }
virtual int handle_data (ACE_SOCK_Stream *logging_peer) { // Ensure blocking <recv>s. logging_peer->disable (ACE_NONBLOCK); ACE_FILE_IO log_file; make_log_file (log_file, logging_peer); Logging_Handler logging_handler (log_file, *logging_peer); while (logging_handler.log_record () != -1) continue; log_file.close (); return 0; handle_data() method similar to before (uses blocking I/O) }
171

class Logging_Process : public ACE_Process { Logging_Process class extends ACE_Process private: Logging_Process (); // Force desired constructor to be used. std::string prog_name_; ACE_SOCK_Stream logging_peer_; public: Logging_Process (const std::string &prog_name, const ACE_SOCK_Stream &logging_peer) : logging_peer_ (logging_peer.get_handle ()) { prog_name_ (prog_name); } prepare() hook method is invoked by spawn() prior to the actual spawning virtual int prepare (ACE_Process_Options &options) { if (options.pass_handle (logging_peer_.get_handle ()) == -1) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "pass_handle()"), -1); pass_handle() adds +H <handle> to command line on Windows options.command_line ("%s", prog_name_.c_str ()); options.avoid_zombies (1); Dont exec() after fork() on POSIX options.creation_flags (ACE_Process_Options::NO_EXEC); return 0; 172 }

virtual void unmanage () { Invoked by process manager as it removes the delete this; managed process }
static void sigterm_handler (int /* signum */) { /* No-op. */ }

int main (int argc, char *argv[]) { // Register to receive the <SIGTERM> signal. ACE_Sig_Action sa (sigterm_handler, SIGTERM); Process_Per_Connection_Logging_Server server; if (server.run (argc, argv) == -1 && errno != EINTR) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "server.run()"), 1); // Barrier synchronization. return ACE_Process_Manager::instance ()->wait (); }
173
The ACE Threading Wrapper Facades

Multithreading is useful for servers that manage connection-oriented or connectionless network associations for many clients simultaneously Today's powerful OS support for multithreading helps networked applications to: Leverage hardware advances, such as symmetric multiprocessing that enables true execution parallelism Increase performance by overlapping computation & communication Improve response time for GUIs & network servers to ensure that time-sensitive tasks are scheduled as needed Simplify program structure by enabling applications to use intuitive synchronous programming mechanisms, rather than more complex asynchronous programming mechanisms Below we describe the following ACE classes that networked applications can use to spawn & manage one or more threads of control within a process:
174
The ACE_Thread_Manager Class (1/2)

Motivation Different operating systems use different APIs to create & manage threads, which causes two types of non-portable variability: Syntactic, e.g., the Win32 CreateThread() & the Pthreads pthread_create() functions provide similar thread creation capabilities, even though their APIs differ syntactically Semantic, e.g., both Pthreads and UI threads support detached threads, whereas Win32 does not, & VxWorks supports only detached threads
175
The ACE_Thread_Manager Class (2/2)

Class Capabilities This class uses the Wrapper Facade pattern to encapsulate variation among different OS multithreading APIs & provide the following portable capabilities: Spawns one or more threads, each running an applicationdesignated function concurrently Alters the most common thread attributes, for example, scheduling priority & stack size, for each of the spawned threads Spawns & manages a set of threads as a cohesive collection, called a thread group Manages the threads in an ACE_Task (discussed in C++NPv2 tutorial)
Facilitates cooperative cancelation of threads, 176 Waits for one or more threads to exit
ACE_Thread_Manager::spawn() Options
The ACE_Thread_Manager::spawn() method can be passed a set of flags to specify the properties of the created thread, whose value is a bitwise inclusive ``or'' of the flags shown in the following table:
177
Thread-per-Connection Logging Server

This example illustrates the thread-per-connection concurrency model The master thread runs continuously and plays the role of a factory that 1.Accepts a connection & creates an ACE_SOCK_Stream object dynamically 2.Spawns a worker thread that uses this object to handle this client's logging The worker thread performs all subsequent log record processing on the ACE_SOCK_Stream & destroys it when the connection is closed
Thread_Per_ Connection_ Logging_ Server
178
Sidebar: Traps & Pitfalls of Mixing Objects & Threads

Context Lifecycle of objects passed as parameters to thread entry point functions must be managed carefully
Problem
Programmers forget to make the following distinction A thread is a unit of execution An object is a chunk of memory with associated methods There is no implicit connection between a thread & any object that the thread accesses during execution Forces
A thread should not be able access an object after the latter has been deleted
Solution Dynamically allocate the object passing it to the thread function as parameter & let the latter delete it
179
Sidebar: How Threads are Spawned in ACE

The figure below show the calls that occur when ACE_Thread_Manager:: spawn() is invoked on a platform configuration that uses the UI Threads thr_create() system function:
Regardless of OS, the following steps occur to spawn a thread: The OS creates a thread execution context The OS allocates memory for the thread's stack The new thread's register set is prepared so that when it's scheduled into execution, it will call the thread entry point function supplied as a parameter to spawn() The thread is marked runnable so the OS can start executing it
180
Using ACE_Thread_Manager (1/5)

This example uses the ACE_Thread_Manager to implement our first multithreaded logging server based on a thread-per-connection concurrency model
class Thread_Per_Connection_Logging_Server : public Logging_Server { private: class Thread_Args { public: Thread_Args (Thread_Per_Connection_Logging_Server *lsp) : this_ (lsp) {} Thread_Per_Connection_Logging_Server *this_; ACE_SOCK_Stream logging_peer_; }; // Passed as a parameter to <ACE_Thread_Manager::spawn>. static void *run_svc (void *arg); protected: // Other methods shown below... };
181

virtual int handle_connections () { auto_ptr<Thread_Args> thread_args (new Thread_Args (this)); if (acceptor ().accept (thread_args->logging_peer_) == -1) return -1; if (ACE_Thread_Manager::instance ()->spawn ( // Pointer-to-function entry point. Thread_Per_Connection_Logging_Server::run_svc, // <run_svc> parameter. Arguments to pass to thread entry point function ACE_static_cast (void *, thread_args.get ()), Create a detached kernel thread THR_DETACHED | THR_SCOPE_SYSTEM) == -1) return -1; thread_args.release (); // Spawned thread now owns memory return 0; }
182

Thread entry point adapter method that handles incoming data from peer
This method will also clean up resources, such as closing the peer connection
void *Thread_Per_Connection_Logging_Server::run_svc (void *arg) { auto_ptr<Thread_Args> thread_args (ACE_static_cast (Thread_Args *, arg)); Were responsible for deleting this dynamically allocated memory when were done, so we use an auto_ptr thread_args->this_->handle_data (&thread_args->logging_peer_); thread_args->logging_peer_.close (); return 0; // Return value is ignored. }
183

virtual int handle_data (ACE_SOCK_Stream *client) { ACE_FILE_IO log_file; // Client's hostname used as logfile name. make_log_file (log_file, client); // Place the connection into blocking mode. client->disable (ACE_NONBLOCK); Logging_Handler logging_handler (log_file, *client); ACE_Thread_Manager *tm = ACE_Thread_Manager::instance (); ACE_thread_t me = ACE_OS::thr_self (); Note the use of the Singleton pattern
// Keep handling log records until client closes connection // or this thread is asked to cancel itself. while (!tm->testcancel (me) && logging_handler.log_record () != -1) continue; log_file.close (); return 0;
}
184
Note the use of cooperative thread cancellation
Sidebar: Serializing Singletons in ACE

As discussed earlier, the Singleton pattern ensures that a class has only one instance & provides a global point of access to it ACE singletons use the Double-Checked Locking Optimization pattern to reduce contention and synchronization overhead when critical sections of code must acquire locks in a thread-safe manner just once during program execution ACE_Thread_Manager *ACE_Thread_Manager::instance () { if (ACE_Thread_Manager::thr_mgr_ == 0) {
ACE_GUARD_RETURN (ACE_Recursive_Thread_Mutex, ace_mon,

*ACE_Static_Object_Lock::instance(), 0)); if (ACE_Thread_Manager::thr_mgr_ == 0)
ACE_Thread_Manager::thr_mgr_ = new ACE_Thread_Manager;

} return ACE_Thread_Manager::thr_mgr_; } ACE_Static_Object_Lock instance is created prior to execution of main() program
185

Thread-per-connection logging server main program int main (int argc, char *argv[]) { // Register to receive the <SIGTERM> signal. ACE_Sig_Action sa (sigterm_handler, SIGTERM); Thread_Per_Connection_Logging_Server server; if (server.run (argc, argv) == -1) ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "server.run()"), 1); // Cooperative thread cancelation. ACE_Thread_Manager::instance ()->cancel_all (); // Barrier synchronization, wait no more than a minute. ACE_Time_Value timeout (ACE_OS::gettimeofday ()); timeout += 60; return ACE_Thread_Manager::instance ()->wait (&timeout); }
186
The ACE_Sched_Params Class (1/2)

Motivation Certain types of networked applications, particularly those with real-time requirements, need strict control over their thread priorities Its important that this control be as portable as possible
187
The ACE_Sched_Params Class (2/2)

Class Capabilities This class uses the Wrapper Facade pattern to encapsulate OS scheduling class APIs & can be used with the ACE_OS::sched_params() method to provide the following capabilities: A portable means to specify scheduling policies, such as FIFO A way to specify a time-slice quantum for the round-robin scheduling policy A way to specify the scope in which the policy applies, for example, to the current process or the current thread
A consistent representation of scheduling priorities, which is necessary since higher scheduling priorities are indicated by lower priority values on some OS platforms 188
Using ACE_Sched_Params (1/2)

This example demonstrates a real-time thread-per-connection logging server whose scheduling parameters can be controlled via ACE_Sched_Params
class RT_Thread_Per_Connection_Logging_Server : public Thread_Per_Connection_Logging_Server { public: virtual int open (u_short port) { ACE_Sched_Params fifo_sched_params (ACE_SCHED_FIFO, ACE_Sched_Params::priority_min (ACE_SCHED_FIFO), ACE_SCOPE_PROCESS); if (ACE_OS::sched_params (fifo_sched_params) == -1) { if (errno == EPERM || errno == ENOTSUP) ACE_DEBUG ((LM_DEBUG, "Warning: user's not superuser, so " "we'll run in the time-shared class\n")); else
189
Using ACE_Sched_Params (2/2)

ACE_ERROR_RETURN ((LM_ERROR, "%p\n", "ACE_OS::sched_params()"), -1);
} // Initialize the parent classes. return Thread_Per_Connection_Logging_Server::open (port);
}
virtual int handle_data (ACE_SOCK_Stream *logging_client) { int prio = ACE_Sched_Params::next_priority (ACE_SCHED_FIFO, ACE_Sched_Params::priority_min (ACE_SCHED_FIFO), ACE_SCOPE_THREAD); Data handling is done at a higher priority than connections ACE_OS::thr_setprio (prio); return Thread_Per_Connection_Logging_Server::handle_data (logging_client); } };
190
The ACE_TSS Class (1/2)

Motivation While C++ global variables can be useful, their potential for harmful side-effects & undefined initialization semantics are problematic These problems are exacerbated in multithreaded applications, e.g., when multiple threads access unsynchronized global variables simultaneously The use of synchronizers in these situations can often cause more problems than they solve The classic example is errno
// ... if (recv () == -1 && errno != EWOULDBLOCK) ACE_ERROR ((LM_ERROR, recv failed, %p, errno)); };
191
The ACE_TSS Class (2/2)

Class Capabilities This class implements the Thread-Specific Storage pattern, which encapsulates & enhances the native OS Thread-Specific Storage (TSS) APIs to provide the following capabilities: It supports data that are ``physically'' thread specific, that is, private to a thread, but allows them to be accessed as though they were ``logically'' global to a program It uses the C++ delegation operator, operator->(), to provide thread-specific smart pointers It encapsulates the management of the keys associated with TSS objects
For platforms that lack adequate TSS support natively (such as VxWorks) ACE_TSS emulates TSS efficiently
192
The Thread-Specific Storage Pattern

The Thread-Specific Storage pattern allows multiple threads to use one logically global access point to retrieve an object that is local to a thread, without incurring locking overhead on each object access
Thread-Specific Object Set
thread 1 key 1 manages The application thread identifier, thread-specific object set, & proxy cooperate to obtain the correct thread-specific object
thread m
Thread-Specific Object Proxy
accesses key n
[k,t]
Thread-Specific Object
: Application Thread
: Thread-Specific Object Proxy

method()
: Key Factory
: Thread-Specific Object Set
create_key()
key
: Thread-Specific Object
TSObject key set()
193
Using ACE_TSS (1/3)

This example illustrates how to implement & apply ACE_TSS to our thread-per-connection logging server In this implementation, each thread gets its own request count that resides in thread-specific storage to alleviate race conditions on the request count without requiring a mutex
template <class TYPE> TYPE * ACE_TSS<TYPE>::operator-> () { if (once_ == 0) { // Ensure that we're serialized. ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, keylock_, 0);
if (once_ == 0) { ACE_OS::thr_keycreate (&key_, &ACE_TSS<TYPE>::cleanup); once_ = 1; } } We used the double-checked locking optimization pattern here
194
Using ACE_TSS (2/3)

TYPE *ts_obj = 0;
// Initialize <ts_obj> from thread-specific storage. ACE_OS::thr_getspecific (key_, (void **) &ts_obj);
// Check if this method's been called in this thread. if (ts_obj == 0) { // Allocate memory off the heap and store it in a pointer. ts_obj = new TYPE; // Store the dynamically allocated pointer in TSS. ACE_OS::thr_setspecific (key_, ts_obj); } return ts_obj; }
195
Using ACE_TSS (3/3)

class Request_Count { public: Request_Count (): count_ (0) {} void increment () { ++count_; } int value () const { return count_; } private: int count_; }; static ACE_TSS<Request_Count> request_count; virtual int handle_data (ACE_SOCK_Stream *) { while (logging_handler_.log_record () != -1) // Keep track of number of requests. request_count->increment (); This call increments variable in thread-specific storage ACE_DEBUG ((LM_DEBUG, "request_count = %d\n", request_count->value ()));
}
196
The ACE Synchronization Wrapper Facades

Different operating systems provide different synchronization mechanisms with different semantics using different APIs Some of these APIs conform to international standards, such as Pthreads Other APIs conform to de facto standards, such as Win32 Below we describe the following ACE classes that networked applications can use to synchronize threads and/or processes portably
197
The ACE_Lock* Pseudo-Class

The ACE mutex, readers/writer, semaphore, & file lock mechanisms all support the ACE_LOCK* interface shown below
ACE_LOCK* is a pseudo-class, i.e., it's not a real C++ class in ACE

We use it to illustrate the uniformity of the signatures supported by many of the ACE synchronization classes e.g., ACE_Thread_Mutex, ACE_Process_Mutex, & ACE_Thread_Semaphore
198
The ACE_Guard Classes

Motivation When acquiring and releasing locks explicitly, it can be hard to ensure that all paths through the code release the lock, especially when C++ exceptions are thrown ACE provides the ACE_Guard class & its associated subclasses to help assure that locks are acquired & released properly
Class Capabilities These classes implement the Scoped Locking idiom, which leverages the semantics of C++ class constructors & destructors to ensure a lock is acquired & released automatically upon entry to and exit from a block of C++ code, respectively
199
The Scoped Locking Idiom

Motivation Code that shouldnt execute concurrently must be protected by some type of lock that is acquired/released when control enters/leaves a critical section If programmers must acquire & release locks explicitly, it is hard to ensure that the locks are released in all paths through the code e.g., in C++ control can leave a scope due to a return, break, continue, or goto statement, as well as from an unhandled exception being propagated out of the scope
void method () { lock_.acquire (); // The implementation may return prematurely lock_.release (); }
The Scoped Locking idiom defines a guard class whose constructor automatically acquires a lock when control enters a scope & whose destructor automatically releases the lock when control leaves the scope
void method () { ACE_Guard <ACE_Thread_Mutex> guard (lock_); // The lock is released when the method returns }
200
Implementing Scoped Locking in ACE

template <class LOCK> class ACE_Guard { Generic ACE_Guard Wrapper Facade public: // Store a pointer to the lock and acquire the lock. ACE_Guard (LOCK &lock) : lock_ (&lock) { lock_->acquire (); } // Release the lock when the guard goes out of scope, ~ACE_Guard () { lock_->release (); }
// Other methods omitted

private: // Pointer to the lock were managing. LOCK *lock_; };
ACE_Write_Guard & ACE_Read_Guard acquire write locks & read locks, respectively
Instances of the ACE_Guard* classes can be allocated on the run-time stack to acquire & release locks in method or block scopes that define critical sections
201
Sidebar: Overview of ACE_GUARD Macros

ACE provides macros to simply the use of the different ACE_Guard classes These macros check for deadlock and failures in lock operations
# define ACE_GUARD(MUTEX,OBJ,LOCK) \ ACE_Guard< MUTEX > OBJ (LOCK); \ if (OBJ.locked () == 0) return; # define ACE_GUARD_RETURN(MUTEX,OBJ,LOCK,RETURN) \ ACE_Guard< MUTEX > OBJ (LOCK); \ if (OBJ.locked () == 0) return RETURN; # define ACE_WRITE_GUARD(MUTEX,OBJ,LOCK) \ ACE_Write_Guard< MUTEX > OBJ (LOCK); \ if (OBJ.locked () == 0) return; # define ACE_WRITE_GUARD_RETURN(MUTEX,OBJ,LOCK,RETURN) \ ACE_Write_Guard< MUTEX > OBJ (LOCK); \ if (OBJ.locked () == 0) return RETURN; # define ACE_READ_GUARD(MUTEX,OBJ,LOCK) \ ACE_Read_Guard< MUTEX > OBJ (LOCK); \ if (OBJ.locked () == 0) return; # define ACE_READ_GUARD_RETURN(MUTEX,OBJ,LOCK,RETURN) \ ACE_Read_Guard< MUTEX > OBJ (LOCK); \ if (OBJ.locked () == 0) return RETURN;
202
The ACE Mutex Classes

Motivation Most operating systems provide some form of mutex mechanism that concurrent applications can use to serialize access to shared resources As with most of the other platform-specific capabilities, there are subtle variations in syntax & semantics between different OS platforms Class Capabilities ACE uses the Wrapper Facade pattern to guide the encapsulation of native OS mutex synchronization mechanisms with the ACE_Process_Mutex & ACE_Thread_Mutex classes These classes implement nonrecursive mutex semantics portably at system scope & process scope, respectively They can be used to serialize thread access to critical sections across processes or in one process The interface for the these classes is identical to the ACE_LOCK* pseudo-class The ACE_Null_Mutex class implements all of its methods as ``no-op'' inline functions
Mutexes also have different initialization requirements
203
Using ACE_Thread_Mutex (1/3)

#include "ace/Synch.h" typedef u_long COUNTER; static COUNTER request_count; // File scope global variable. // Mutex protecting request_count (constructor initializes). static ACE_Thread_Mutex m; virtual int handle_data (ACE_SOCK_Stream *) { while (logging_handler_.log_record () != -1) { // Try to acquire the lock. if (m.acquire () == -1) return 0; ++request_count; // Count # of requests m.release (); // Release lock. This code is tedious } & error-prone to m.acquire (); write & maintain! int count = request_count; m.release (); ACE_DEBUG ((LM_DEBUG, "request_count = %d\n", count)); logging_handler_.close (); return 0; }
204

handle_data() method shown using ACE_GUARD macros virtual int handle_data (ACE_SOCK_Stream *) { while (logging_handler_.log_record () != -1) { // Acquire lock in constructor. ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, m, -1); ++request_count; // Count # of requests // Release lock in destructor. Obtrusive m must be initialized properly (C++ } change does not guarantee any order) int count; Source of accidental complexity { ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, m, -1); count = request_count; } ACE_DEBUG ((LM_DEBUG, "request_count = %d\n", count)); }
205

Ensuring consistent initialization order of global & static members using ACE_Object_Manager
// ... while (logging_handler_.log_record () != -1) { // Acquire lock in constructor. ACE_GUARD_RETURN (ACE_Recursive_Thread_Mutex, guard, ACE_Static_Object_Lock::instance (), -1); ++request_count; // Count # of requests // Release lock in destructor. } // ... ACE_Static_Object_Lock gets preinitialized before main() runs
206
The ACE Readers/Writer Classes

Motivation
Readers/writer locks allow efficient concurrent access to resources whose contents are searched much more often than they are changed Operating systems support readers/writer semantics in their file-locking APIs Involving the file system in synchronization activities is unnecessarily inefficient, however, & can block under unpredictable situations Class Capabilities ACE encapsulates the native readers/writer lock mechanisms with the ACE_RW_Thread_Mutex & ACE_RW_Process_Mutex classes. These classes apply the Wrapper Facade pattern to implement the semantics of process- and systemscoped readers/writer locks portably The interface for these classes is identical to the signatures of ACE_LOCK* pseudo-class The ACE readers/writer implementation gives preference to writers, i.e., if there are multiple readers and a single writer waiting on the lock, the writer will acquire it first
Moreover, file-locking mechanisms work only at the system-scope level, rather than at process scope
207
Using ACE_RW_Thread_Mutex (1/3)

Implementing atomic operations using reader/writer locks class Atomic_Op { public: // Initialize <count_> to <count>. Atomic_Op (long count = 0) : count_ (count) {}
// Atomically pre-increment <count_>. long operator++ () { // Use the <acquire_write> method to acquire a write lock. ACE_WRITE_GUARD_RETURN (ACE_RW_Thread_Mutex, guard, lock_, -1); return ++count_; }
Note the use of the Scoped Locking idiom
ACE provides the ACE_Atomic_Op<> template thats based on this approach.

208

// Atomically return <count_>. operator long () { // Use the <acquire_read> method to acquire a read lock. ACE_READ_GUARD_RETURN (ACE_RW_Thread_Mutex, guard, lock_, 0); Multiple threads can return count_; be reading this value } concurrently // ... Other arithmetic operators omitted. private: // Readers/writer lock. ACE_RW_Thread_Mutex lock_; // Value of the <Atomic_Op> count. long count_; };
209

Overcoming the need for obtrusive changes using class Atomic_Op typedef Atomic_Op COUNTER;
static COUNTER request_count; // File scope global variable.

virtual int handle_data (ACE_SOCK_Stream *) { while (logging_handler_.log_record () != -1) // Keep track of number of requests. ++request_count; // Actually calls <Atomic_Op::operator++>. ACE_DEBUG ((LM_DEBUG, "request_count = %d\n", // Actually calls <Atomic_Op::operator long>. (long) request_count)); }
210
The ACE Semaphore Classes

Motivation Class Capabilities
Semaphores are a powerful mechanism used to lock and/or synchronize access to shared resources in concurrent applications
A semaphore contains a count that indicates the status of a shared resource Application designers assign the meaning of the semaphore's count, as well as its initial value Semaphores can therefore be used to mediate access to a pool of resources
211
The ACE_Thread_Semaphore & ACE_Process_Semaphore classes portably encapsulate process-scoped & system-scoped semaphores, respectively, in accordance with the Wrapper Faade pattern
These class interfaces are largely the same as the ACE_LOCK* pseudo-class The ACE_Null_Semaphore class implements all of its methods as ``no-op'' inline functions
Using ACE_Thread_Semaphore (1/8)

class Message_Queue Message queue implementation using { ACE_Thread_Semaphore public: // Default high and low water marks. enum { DEFAULT_LWM = 0, // 0 is the low water mark. DEFAULT_HWM = 16 * 1024 // 16 K is the high water mark. }; // Initialize. Message_Queue (size_t = DEFAULT_HWM, size_t = DEFAULT_LWM); // Destroy. ~Message_Queue (); // Checks if queue is full/empty. int is_full () const; int is_empty () const;
212

// Interface for enqueueing and dequeueing ACE_Message_Blocks. int enqueue_tail (ACE_Message_Block *, ACE_Time_Value * = 0); int dequeue_head (ACE_Message_Block *&, ACE_Time_Value * = 0);
Note use of Thread-Safe Interface pattern
private: // Implementations that enqueue/dequeue ACE_Message_Blocks. int enqueue_tail_i (ACE_Message_Block *, ACE_Time_Value * = 0); int dequeue_head_i (ACE_Message_Block *&, ACE_Time_Value * = 0);
// Implement the checks for boundary conditions. int is_empty_i () const; int is_full_i () const; // Lowest number before unblocking occurs. int low_water_mark_; // Greatest number of bytes before blocking. int high_water_mark_;
213
Thread-Safe Interface Pattern

Context Components in multi-threaded applications that contain intracomponent method calls Problem Thread-safe components should be designed to avoid unnecessary locking Thread-safe components should be designed to avoid self-deadlock Solution Apply the Thread-safe Interface design pattern (P2) to minimize locking overhead & ensure that intracomponent method calls do not incur self-deadlock by trying to reacquire a lock that is held by the component 214 already This pattern structures all components that process intra-component method invocations according two design conventions: Interface methods check All interface methods, such as C++ public methods, should only acquire/release component lock(s), thereby performing synchronization checks at the border of the component. Implementation methods trust Implementation methods, such as C++ private and protected methods, should only perform work when called by interface methods.

// Current number of bytes in the queue. int cur_bytes_; // Current number of messages in the queue. int cur_count_; // Number of threads waiting to dequeue a message. size_t dequeue_waiters_; // Number of threads waiting to enqueue a message. size_t enqueue_waiters_;
// C++ wrapper facades to coordinate concurrent access. mutable ACE_Thread_Mutex lock_; ACE_Thread_Semaphore notempty_; ACE_Thread_Semaphore notfull_; Using ACE_Thread_Semaphore // Remaining details of queue implementation omitted....
};
215

Message_Queue::Message_Queue (size_t hwm, size_t lwm) : low_water_mark_ (lwm), high_water_mark (hwm), cur_bytes_ (0), cur_count_ (0), dequeue_waiters_ (0), enqueue_waiters_ (0), notempty_ (0), notfull_ (1) { /* Remaining constructor implementation omitted ... */ } int Message_Queue::is_empty () const { ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, lock_, -1); return is_empty_i (); } int Message_Queue::is_full () const { ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, lock_, -1); return is_full_i (); }
216

int Message_Queue::is_empty_i () const { return cur_bytes_ <= 0 && cur_count_ <= 0; } int Message_Queue::is_full_i () const { return cur_bytes_ >= high_water_mark_; } int Message_Queue::enqueue_tail (ACE_Message_Block *new_mblk, ACE_Time_Value *timeout) { ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, lock_, -1); int result = 0; // Wait until the queue is no longer full. while (is_full_i () && result != -1) { ++enqueue_waiters_; guard.release (); result = notfull_.acquire (timeout); guard.acquire (); }
217

if (result == -1) { if (enqueue_waiters_ > 0) --enqueue_waiters_; if (errno == ETIME) errno = EWOULDBLOCK; return -1; } // Enqueue the message at the tail of the queue. int queued_messages = enqueue_tail_i (new_mblk); // Tell any blocked threads that the queue has a new item! if (dequeue_waiters_ > 0) { --dequeue_waiters_; notempty_.release (); } return queued_messages; // guard's destructor releases lock_.
218

int Message_Queue::dequeue_head (ACE_Message_Block *&first_item, ACE_Time_Value *timeout) { ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, lock_, -1); int result = 0;
// Wait until the queue is no longer empty. while (is_empty_i () && result != -1) { ++dequeue_waiters_; guard.release (); result = notempty_.acquire (timeout); guard.acquire (); }
if (result == -1) { if (dequeue_waiters_ > 0) --dequeue_waiters_; if (errno == ETIME) errno = EWOULDBLOCK; return -1; }
219
// Remove the first message from the queue. int queued_messages = dequeue_head_i (first_item);
// Only signal if we've fallen below the low water mark. if (cur_bytes_ <= low_water_mark_ && enqueue_waiters_ > 0) { enqueue_waiters_--; notfull_.release (); } return queued_messages; // <guard> destructor releases <lock_> }
220
ACE Condition Variable Classes (1/2)

Motivation Condition variables allow threads to coordinate & schedule their processing efficiently Condition variables are more appropriate than mutexes or semaphores when complex condition expressions or scheduling behaviors are needed e.g., condition variables are often used to implement synchronized message queues that provide producer/consumer communication to pass messages between threads
Producer <<put>> Thread Request Queue
put() get()
<<get>>
Consumer Thread
uses 2
uses
ACE_Thread_Condition wait() signal() broadcast()

221
ACE_Thread_Mutex
acquire() release()
ACE Condition Variable Classes (2/2)

Class Capabilities The ACE_Condition_Thread_Mutex uses the Wrapper Faade pattern to guide its encapsulation of process-scoped condition variable semantics The ACE_Null_Condition is a zero-cost class whose interface conforms to the ACE_Condition_Thread_Mutex
222
Using ACE_Condition_Thread_Mutex (1/3)

ACE_Recursive_Thread_Mutex class definition Class ACE_Recursive_Thread_Mutex { public: ACE_Recursive_Thread_Mutex (const char *name, void *arg); int acquire (); int release (); // ... private: Demonstrating use of int nesting_level_; ACE condition variable ACE_thread_t owner_id_; wrapper facade ACE_Thread_Mutex lock_; ACE_Condition_Thread_Mutex lock_available_; }; ACE also has an ACE_Condition_Recursive_Thread_Mutex class!
223

ACE_Recursive_Thread_Mutex::ACE_Recursive_Thread_Mutex (const char *name, void *arg) : nesting_level_ (0), owner_id_ (0), lock_ (name, arg), // Initialize the condition variable. lock_available_ (lock_, name, arg) { }
int ACE_Recursive_Thread_Mutex::release () { // Automatically acquire mutex. ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, lock_, -1); nesting_level_--; if (nesting_level_ == 0) { lock_available_.signal (); // Inform waiters the lock is free. owner_id_ = 0; } return 0; // Destructor of <guard> releases the <lock_>. }
224
Demonstrating use of condition variables

1 int ACE_Recursive_Thread_Mutex::acquire () 2 { 3 ACE_thread_t t_id = ACE_OS::thr_self (); 4 5 ACE_GUARD_RETURN (ACE_Thread_Mutex, guard, lock_, -1); 6 7 if (nesting_level_ == 0) { 8 owner_id_ = t_id; 9 nesting_level_ = 1; 10 } 11 else if (t_id == owner_id_) 12 nesting_level_++; 13 else { Demonstrating use of 14 while (nesting_level_ > 0) condition variables 15 lock_available_.wait (); 16 17 owner_id_ = t_id; 18 nesting_level_ = 1; 19 } 20 return 0; 21 }
225
Additional Information
Patterns & frameworks for concurrent & networked objects
www.posa.uci.edu ACE & TAO open-source middleware www.cs.wustl.edu/~schmidt/ACE.html www.cs.wustl.edu/~schmidt/TAO.html
ACE research papers www.cs.wustl.edu/~schmidt/ACE-papers.html Extended ACE & TAO tutorials UCLA extension, July 2004 www.cs.wustl.edu/~schmidt/UCLA.html ACE books www.cs.wustl.edu/~schmidt/ACE/ 226

C NPv1

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

C NPv1

Uploaded by

Copyright:

Available Formats

C++ Network Programming

Mastering Complexity with ACE & Patterns

Motivation: Challenges of Networked Applications

The Layered Architecture of ACE

Large open-source user community www.cs.wustl.edu/~schmidt/ACE-users.html 14

Commercial support by Riverace www.riverace.com/

Sidebar: Platforms Supported by ACE

Key Capabilities Provided by ACE

The Pattern Language for ACE

The Frameworks in ACE

ACE Framework Reactor & Proactor Service Configurator Task Acceptor-Connector

Networked Logging Service Example

Patterns in the Networked Logging Service

Pipes & Filters AcceptorConnector Component Configurator Proactor Wrapper Facade

ACE Basics: Logging

ACE Logging Usage

Displays the current time

MASK_TYPE is ACE_Log_Msg::PROCESS or ACE_Log_Msg::THREAD.

Logging Severities Example

ACE_Thread_Manager::instance ()->spawn (service);

Redirect Logging to a File

Redirect Logging to Syslog

Useful Logging Flags

Networked Logging Service Example

Well develop architecture similar to ACEs, but not same implementation.

Network Daemon Design Dimensions

Communication Design Dimensions

Connectionless vs. Connection-oriented Protocols

3-way handshake in TCP/IP

Alternative Connection Muxing Strategies

Sync vs. Async Message Exchange

Message Passing vs. Shared Memory

Message passing exchanges data explicitly via the IPC mechanisms

Sidebar: C++ Objects & Shared Memory

General responsibilities using placement new operator

Overview of the Socket API (1/2)

Overview of the Socket API (2/2)

Connection establishment & termination

Data transfer mechanisms

Options management Network addressing

Taxonomy of Socket Dimensions

Communication & connection role

Limitations with the Socket APIs (1/2)

API is linear rather than hierarchical

No consistency among the names Non-portable & error-prone

Limitations with the Socket APIs (2/2)

Steep learning curve due to complex semantics

Too many low-level details

Example of Socket API Limitations (1/3)

// Create the local endpoint.

Example of Socket API Limitations (2/3)

Use of non-portable return value

Wrong byte order

addr.sin_addr.addr = INADDR_ANY; Unused structure members not zeroed out

Example of Socket API Limitations (3/3)

SOCK_DGRAM handle illegal here

No guarantee that n bytes will be written close (n_handle);

ACE Socket Wrapper Faade Classes

The Wrapper Faade Pattern (1/2)

The Wrapper Faade Pattern (2/2)

data method1() methodN()

void method1(){ functionA(); functionB(); }