Complex Event Processing

Framework for Operational Visibility and Decisions

COMPLEX EVENT PROCESSING

TABLE OF CONTENTS

1 Introduction ..........................................................................3 2 CEP: Concepts ..........................................................................4 3 CEP: Challenges.......................................................................7 4 TIBCO BusinessEvents: CEP Software. ...............................8 5 Relationship to Other technology Investments .... 11 6 Conclusion ............................................................................13 7 about tibco . ..........................................................................14

COMPLEX EVENT PROCESSING

1. Introduction
Surprise is rarely a good thing in business. Take a look at some of the surprises from the last five years: the Enron scandal, the Worldcom bankruptcy, restatement of revenues from multinational corporations, massive financial fraud, earnings shortfalls, the iPod success and the list goes on and on. These unexpected developments range in effect from inconvenient to disastrous or, in the case of pleasant surprises, opportunities. As a result of these developments, there is an increased emphasis on corporate governance and regulatory compliance requirements such as the Sarbanes-Oxley Act of 2002, the Patriot Act, the Basel II or the Container Security Initiative (CSI). Now more than ever, CEOs, CFOs and other decision makers must be connected to the everyday events and occurrences throughout their enterprises. These enterprises, in turn, require real-time insight and a reporting infrastructure for all decision makers and/or decision enablers. There are at least two challenges in meeting this requirement. First, most IT infrastructures cannot handle the volume of events generated as part of daily operations in real-time. Second, most IT infrastructures are incapable of maintaining relationships among discrete granular events from various IT and business layers across the enterprise and correlating those events with historical context. The first challenge is addressed by using TIBCOs industry-leading real-time infrastructure. Since the late 1980s, TIBCO has pioneered real-time business, starting with the integration and delivery of market data on the trading floors of large banks and financial services institutions. The second challenge is addressed by complex event processing (CEP). TIBCO BusinessEvents provides comprehensive CEP capabilities for real-time operational insight enabling timely, well-informed decisions. This technology lets organizations manage risk more effectively and prevents industry dissonance when market realities outpace corporate strategies by making smarter, well-informed operational decisions every minute of every day. This paper reveals the concepts, applications and technology behind TIBCOs CEP offering.

COMPLEX EVENT PROCESSING

2. CEP: Concepts
Complex event processing (CEP) is by no means a revolutionary concept. In our lives, we apply CEP to many facets of our routine, whether it is driving to work where we are reacting to traffic flow changes or at work where we are accommodating changing project deadlines and dealing with unexpected circumstances. In each case, we attempt to interpret the events of everyday life in the context of our current understanding of the way things should be. This concept of should be is really a conscious attempt to pattern an acceptable scenario where we avoid unpleasant surprises or take advantage of fortuitous opportunities. The constant evaluation of singular or multiple occurrences (events) in the context of expected behavior and acceptable parameters is the basis of realtime decision-making and scenario prediction. This correlation of events, current and past, is an essential decision analysis component of the human brain. CEP introduces this same mechanism to electronically capture known event patterns for automating part of the decision making process in an enterprise. Prior to introducing the concepts of CEP, it would be worthwhile to illustrate a few examples of what CEP can do: A technician is dispatched to a customer location and the customer cancels the order. At the same time, another customer in a nearby location places an order. The technician is dynamically re-routed to the new customer. A frequent traveler, who almost always requests hotel room upgrades, checks into a hotel and forgets to ask for an upgrade. At the registration desk, the manager says I see that you typically stay in our suites, we have an upgrade to an executive suite available for $20 additional charge. Web site users are proactively provided with shopping and auction menus based on previous navigation events that constitute their specific web site and content behaviors. A brokerage notices unusual call option activity from a brokerage customer who has never traded in options or futures. This unusual activity triggers trading limits based on velocity and magnitude of trades for the brokerage to manage risk and alerts customer service so that they can proactively contact the customer and verify the transaction.

COMPLEX EVENT PROCESSING

Program traders need the flexibility to execute various strategies to capitalize on opportunities. For example, If the stock price of X moves above 2% of the volume weighted average price (VWAP) and stock price of Y moves below 2% of VWAP, then sell 1000 shares of X and buy 5000 shares of Y. The ability to capture real-time events and present those events in context gives traders a better understanding of market changes and the ability to respond quickly to those changes. A pallet of pharmaceuticals is in transit within the logistics supply chain and needs to be recalled for quality control purposes. Radio frequency identification (RFID) location and pallet identification events are tracked and the pallet is intercepted prior to breakdown and retail distribution. A global telecommunications provider is supporting numerous productbundle packages that have unique Quote-to-Cash elements and processes imbedded. As new packages are released, customers dynamically change their selections, options, installation, terms and conditions (including invoked Service Level Agreements) in response to their changing needs. Event-driven processes interactively respond to changed conditions and requirements and reflect appropriate package assembly, deployment and billing to the customer and the service provider. An independent power operator notices peak load changes across the power grid that have the potential to disrupt large portions of the user community. These operational, real-time events are understood to fall into a pattern that is recoverable based on dynamic allocation and load management across the grid. No subsequent outage is experienced.

COMPLEX EVENT PROCESSING

The CPU and memory utilization of a critical application server has been trending upwards for the last 10 minutes. These real-time IT events correspond to potential shutdown of the application server and based on historical context dynamically allocate additional IT resources to prevent disruption of service. There is a common pattern behind these varied problems and opportunities from the business and IT world. These problems represent a new genre of applications: event-driven applications that make automated decisions based on a pattern of events and also provide a rich environment for the analysis of real-time events in combination with the rich historical context from traditional data warehouses. There is very little argument against the relevance or importance of these problems to any enterprise, IT or business. Every business would want to catch or predict exceptions and detect opportunities at the earliest possible moment. What is missing from the current infrastructure is a mechanism to define surprises at an enterprise level. What is missing is a mechanism to define event chains that identify what is normal, compliant or expected. What is missing is a mechanism to instrument and collect events at a granular level not only within the enterprise but across the extended value chain. What is missing is the correlation of real-time event streams with the historical context. An enterprise-ready CEP solution needs to have the following characteristics: Integration ready: As the number of event sources and applications grows with the increased demand for faster execution of processes, organizations are in dire need of new technologies that can help them manage the complexity of their environment. The events organizations have access to are not always tailored to the problems they are trying to solve, said David Luckham, Stanford University Professor and author of The Power of Events. Integration technology is a key to success for developing any CEP application. Scalable: RFID, wireless sensors, web site tracking are all examples of sensor technologies that are at various stages of deployment. Scalability, horizontal and vertical, is a critical requirement in order to handle the high throughput event volumes. Distributed agent-based architecture: In contrast to the hub-and-spoke model, this architecture requires edge agents to have local decision making abilities. In RFID scenarios, the local edge agents correlate the events from

COMPLEX EVENT PROCESSING

multiple readers, detect incorrect reads and forward the legitimate events to the next layer. Real-time reporting and analysis: CEP is primarily focused on detecting modeled patterns in an event stream. One of the challenges is the discovery and evolution of models and patterns. The solution needs to provide a visual environment for the analysis and mining of events for patterns and root causes. Automated attendants: In conjunction with real-time reporting and analysis, the solution should provide an infrastructure to automate decisions once models and patterns are discovered. This is traditionally addressed by EventCondition-Action (ECA) type of rules. Enterprise worthiness: A solutions ability to support a multi-user development environment, version control, and lifecycle management, while often ignored, is the key to success for an enterprise-wide deployment of CEP.

3. CEP: Challenges
EVENTS: ARE THEY AVAILABLE? It is often mentioned that events needed to anticipate surprises or catch opportunities do not exist. As recently as ten years ago this may have been a partially valid objection. The current reality, however, is that the events are there, although they are often not in the hands of the right person. Because of the vast investment in business process reengineering, business process management (BPM) and enterprise application integration (EAI), many organizations know more than ever about their own workflows. They know what each individual is doing in the business process, be it taking an order, compiling a report or driving a rivet. The constant flux of RFID events requires real-time analysis to optimize these processes by using an intelligence layer. In addition, with the advent of RFID-based intelligent sensor networks, the events can be captured with the semantics of time and location at a granular level. The presence of these events, if utilized well, can be used to optimize the business processes at a local and/or a global level. EVENTS: ARE THERE WARNINGS? More often than not, warnings precede threats or opportunities. The important first step is instrumentation of the enterprise to collect and correlate the events with the respective business processes. An application server exhibiting a constantly increasing CPU and/or memory utilization can be interpreted as a

COMPLEX EVENT PROCESSING

warning for the server shutdown. The fact is that this is a known patternone very rarely detected as a warning but more commonly used in an after-the-fact analysis.

4. TIBCO BusinessEvents: CEP Software

TIBCO BusinessEvents provides the following sophisticated CEP capabilities: A model to capture expected outcomes A way of detecting variances using a combination of the presence and/or absence of events A model for responding to the detected variances CONTEXTUAL MODEL FOR CAPTURING EXPECTED OUTCOMES In order to meet or exceed service levels, all of the service participants are expected to operate within known tolerances. A deviation in any component can potentially affect downstream components thus resulting in service disruption. The business rules and constraints required to govern any service according to the service levels, required to capture the exceptions in any process flow, required to dynamically interact with customers based on the known customer behavior are easily represented and maintained using a model driven approach in contrast to a pure rules driven approach. TIBCO BusinessEvents provides a UML compliant modeling framework to capture the static and dynamic relationships between the participants. CONCEPTUAL MODEL: CAPTURE THE STATIC RELATIONSHIPS This model is loosely based on UML class-relationship concepts and provides a mechanism to capture the semantics of static relationships. In a Service Level Management example, application A composed of services S1 and S2, deployed on server X can be represented as four concepts A, S1, S2 and X connected by static relationships is composed of and deployed on. In addition to modeling the semantics of the relationship, the model can capture the transitivity of certain relationships such as depends on or composed of. This is extremely useful in reducing the redundancy of the model. STATE MODEL: CAPTURE THE DYNAMIC RELATIONSHIPS This state model is completely based on UML state-charts and provides a mechanism to capture the semantics of dynamic flows such as a package flow

COMPLEX EVENT PROCESSING

across a network of hubs or an order flow through many internal systems and partners. The state model is a context for modeling the known states and outcomes in various scenarios. The context is created by specifying correlation rules that are a function of the presence and/or absence of events. AGGREGATION MODEL: CAPTURE THE CONTEXT OF RELATED EVENTS Though the state model provides a rich context for the event chains leading up to an exception, an alert or an opportunity, it does not provide the historical context required for operational decisions. For instance in a financial trading example, if a stock dips below its exponential 50-day moving average, then sell the stock. This example requires TIBCO BusinessEvents to maintain or retrieve the aggregate. TIBCO BusinessEvents provides two modes for aggregates: Internal mode: In the internal mode, TIBCO BusinessEvents provides a mechanism to define aggregation rules and populate either static scorecards or dynamically versioned instances. For instance, a rule can be configured to maintain a 50-second moving average of all the stocks in a portfolio. External mode: In the external mode, TIBCO BusinessEvents provides a call out mechanism to external databases and data warehouses as part of the rule execution. For instance, average orders/day for a certain customer can be obtained from the data warehouse and a rule such as Orders for the day < Average orders/day can be applied to the order streams. To summarize: this approach, in contrast to a traditional alerts or rules engine, offers a model-driven framework that collects and tabularizes the events and uses the state model as the basis for writing the constraints discussed below. EVENT MODEL Events are fundamentally a record of an occurrence. An event in its basic form is richer as it carries the time and location of occurrence along with relevant data. Simple events are records of something that has happened and can range from an update to a customer record in the CRM application, a point-of-sale (POS) event, an order being scanned upon loading into a truck or upon delivery to a customer, GPS data for delivery trucks in route, weather information, RFID data for palettes or display stands and more.

10

COMPLEX EVENT PROCESSING

BUSINESS RULES AND CONSTRAINTS MODEL FOR HANDLING VARIANCES Business rules and constraints are the core of a CEP solution and TIBCO BusinessEvents follows a model similar to the Rapide event pattern language.1 The Rapide language is a full-featured events and rules language of the form EventCondition-Action (ECA). The ECA rules are independent units and forward-chained by the engine based on the state changes. The rules supported by TIBCO BusinessEvents can be categorized into two distinct functional categories: Event correlation and aggregation: These types of rules are an essential component of the contextual model. These rules, once executed create a rich contextual model for applying the constraint rules. Constraints: These types of rules are applied to create constraint and threshold boundaries that the engine evaluates based on state changes. The constraint rules can be of many types, some of them being: Linear thresholds: These rules are primarily used as warning indicators for key performance indicators (KPIs). Risk tolerance: The ability to monitor risk with the granularity of a singular event enables a range of real-time mitigation and management techniques not previously achievable with stand-alone applications. Whether the business is monitoring Value at Risk (VaR), Supply Chain Risk or Operational Risk, the employment of a real-time event-enabled platform permits individual components of risk to be isolated, prioritized and structured into a composite picture of enterprise exposure and consequence. The application of dynamic risk element monitoring over time can be attained by using temporal processing in association with event occurrence thereby providing a picture of either degrading or improving risk scenarios. Population of these second order effects into a visualization environment or even into a risk-mitigation action application provides the business with the ability to avoid run-away situations or to dynamically adjust risk tolerance on a case-by-case basis. Algorithmic variances: In certain industries such as financial services, industrial control and interactive marketing, the notion of variable ranges of measurement, tolerance and reaction are important. Moving away from binary decision making mechanisms is important where the priority or

11

COMPLEX EVENT PROCESSING

context of singular events can have a range of influence, meaning or utility. The decisions that are made or the action that is taken with regard to specific event occurrence may need to be flexible and adaptable to external situations and moving criteria. The real-time and granular aspect of an event-enabled platform assures the business that a variety of operation models and scenario response actions can be modeled by taking into account external factors which may influence how a specific outcome should be handled or resolved. Temporal reasoning: These rules are typically predicated on the presence or absence of events and the object states, and are a function of time. Reasoning over moving averages in a sliding window is a typical scenario in which temporal reasoning is required. Spatial reasoning: These rules are typically predicated on the presence or absence of events and the object states, and are a function of location. Reasoning over geometry to infer proximity (near-by, north-of, south-of) is a typical scenario in which spatial reasoning is required.

5. Relationship to Other Technology Investments

HOW DOES CEP RELATE TO BUSINESS INTELLIGENCE? Since the mid to late 1990s, enterprises have realized the importance of integrating and transforming data from disparate applications and separate business units into a data warehouse for reporting, analysis and decision-making purposes. The challenge with a data warehouse-only approach is two fold: Time needed to change Cost of implementing change The need for implementation speed to match the velocity of business, evidenced by the ever changing and increasingly competitive marketplace, is well documented. The only constant in business is change and a data warehouse-only approach to operational decision making is challenged to respond in a timeframe that has relevance to the ever changing tactical scenarios faced by business. How many times has IT implemented a report or a dashboard for business users only to have their insatiable appetite for new information require changes to a KPI or

12

COMPLEX EVENT PROCESSING

a dimension or fact? This is often referred to as information latency, generally characterized as after the fact insight, which prevents any sort of timely action. Given the scope and breadth of the data warehouse charter, they also tend to be complicated and cautiously maintained platforms which lead to an investment dynamic proportionate to the scale of decision making required by the environment. CEP-based solutions provide the link that unites the decision making cycle by providing a layer of timely, relevant and insightful operational information to business users that addresses the multitude of critical scenarios faced daily by business. HOW DOES CEP RELATE TO BUSINESS PROCESSES? No set of mechanisms better exemplifies the generation, interaction and consumption of events than that of an enterprises operational business processes. Where data was once seen as the ultimate of corporate assets, business processes are also now viewed in the same light. However, peel back the covers of any business process and you will see the presence of a number of connected and (seemingly) unconnected events governed by designated timelines and result states. External operational conditions and influences to established business processes and the requirement to support dynamic process instantiation and orchestration are really just questions of managing a series of complex events from a context, sequence and result perspective. With industry acceptance of Event Driven Architectures (EDA), the role of CEP with regard to operational business process will continue to grow and will, in the not distant future, become a design construct for new processes and process performance optimization and granular diagnosis (process forensics). HOW DOES CEP RELATE TO BUSINESS ACTIVITY MONITORING? Business activity monitoring (BAM) aims to improve operational business visibility across systems, people and partners. While BAM has appealed to many enterprises at the departmental level, enterprise level implementation has become extremely complex. CEP provides increased capabilities for managing the patterns of events required to attain various levels of visibility across a spectrum of roles and users throughout an enterprise. CEP manages the models of processes, events and rules at an enterprise level and feeds the various BAM management and presentation layers as needed by the business and operations.

13

COMPLEX EVENT PROCESSING

Too much information can overwhelm a business user. BAM tries to reduce information overload by delivering only the information that users need to perform their tasks and manage the business. This is achieved by presenting situations and events that require action and CEP can help by providing the drill down necessary to conduct both horizontal and vertical root-cause analysis.

6. Conclusion
Businesses face a stream of connected and unconnected operational scenarios each day relating to customer sales, marketing, logistics, supply chain, product development, problem resolution, accounting, billing or support. What is common to each scenario is that specific events characterize the behavior of each mechanism, specific events characterize the corporate resources mobilized/utilized for each mechanism and specific events characterize the business process design and execution for each mechanism. Unlike other technologies that have been employed to support corporate operations through enhanced decision making (e.g. business intelligence and BPM), CEP augments the decision-making tool kit by concentrating on an operational understanding of the business from the following perspectives: Tactical business scenarios Real-time scenario decision making Contextual understanding of the scenario Day-to-day business (operations) is made up of small tactical triumphs and tragedies. Often the decision-making process applied to this environment is unscientific, untimely, reactionary and lacking in broader context and meaning. It is, however, this very same environment that constitutes the major cost and revenue opportunities of the business, and also represents the external character of the business to the marketplace, customers and partners. Providing a decisionmaking capability that addresses tactical (scenario-specific) decisions, at specific inflection points (real-time) and with a level of context (event history, priority and circumstance) is the aim of CEP. These three event-centric conditions together form an operational pattern that allows the scenario to be assessed and either acted

upon reactively or proactively, based on known norms of operation or better still, norms of operational excellence. The end result is a continuous and sustained ability by the business to avoid disruptive and expensive operational problems and/or enable the business to leverage new opportunities.

7. About TIBCO
TIBCO Software Inc. (NASDAQ: TIBX) is a provider of infrastructure software for companies to use on-premise or as part of cloud computing environments. Whether its optimizing claims, processing trades, cross-selling products based on real-time customer behavior, or averting a crisis before it happens, TIBCO provides companies the two-second advantageTM the ability to capture the right information at the right time and act on it preemptively for a competitive advantage. More than 4,000 customers worldwide rely on TIBCO to manage information, decisions, processes and applications in real time. Learn more at www.tibco.com

Global Headquarters 3303 Hillview Avenue Palo Alto, CA 94304

Tel: +1 650-846-1000 +1 800-420-8450 Fax: +1 650-846-1005

www.tibco.com

2011 TIBCO, TIBCO Software, and The Two-Second Advantage are trademarks or registered trademarks of TIBCO Software Inc. and its subsidiaries in the United States and/or other countries. All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only.