TIBCOs Complex Event Processing (CEP) Offering

TIBCO BUSINESS EVENTS

AGENDA
Introduction to CEP & Business Events Business Events Palette Overview Channels & Destinations Events Concepts Rules & Rule Sets Scorecards

COMPLEX EVENT PROCESSING

CEP
basis is a set of technologies that allows events to be processed on a continuous

COMPLEX EVENT PROCESSING

Typical Complex Event Processing Area

HAS
Situation awareness Sense and respond Track and trace

These aspects may overlap in actual business situations

SITUATION AWARENESS
KNOWING the state of product, person, document or entity of interest at any particular time Requires continuous MONITORING of events EXAMPLE The REAL TIME DASHBOARD indicates all the performance indicators for a runtime production process.

SENSE AND RESPOND

DETECTION of significant fact about the product, person, document or entity of interest and RESPOND accordingly To achieve this, SYSTEM performs a) MONITORING OF EVENTS b) DETECTION OF SOMETHING SIGNIFICANT c) EXECUTION OF REQUIRED RESPONSE EXAMPLE Monitor credit card usage and detect that the credit card is being used consecutively at locations that are too far apart for real time person to business transactions. Fraud is detected and the transactions are denied.

TRACK AND TRACE

TRACKING the product, person, document or entity of interest over time TRACING pertinent facts like location, owner or general status

EXAMPLE Tracking events from an RFID-enabled inventory control system where at any point in time you need to know the status of the delivery of goods at a particular location

BUSINESS EVENTS MAJOR COMPONENTS

TIBCO Designer
WORKBENCH TIBCO Administrator Browser

Project Files

STATE MODELER

EAR File

TIBCO Designer RUN

TIBCO Business Events Server TIBCO Administrator WORKBENCH Project Files

Runtime Data Store

Browser

MODEL DRIVEN APPROACH IN BUSINESS EVENTS

TIBCO Business Events enables CEP problems to be solved through a MODEL DRIVEN approach, in which the developer defines the event, rule, concept(class) and state models which are then compiled so that run time incoming events are processed efficiently as possible

Describes INPUTS in BE

Describes BEHAVIORAL MECHANISM in BE

Describes DATA in BE

Describes STATES, TRANSITION & CONDITIONS in BE

AGENDA
Introduction to CEP & Business Events Business Events Palette Overview Channels & Destinations Events Concepts Rules & Rule Sets Scorecards

BUSINESS EVENTS PALETTES

Business Events State Modeler Palette mainly consists of : Business Events Workbench Palette mainly consists of : a) State Machine a) Channel Destination b) Call State Machine b) Concept c) Simple c) Concept View d) Composite d) Simple Event & Time Event e) Concurrent e) Score Card Activities Palette mainly consists of : f) Annotation Business Events f) Rule Set Rule g) End a) Receive Event g) Rule Function b) Send Event c) Wait for Event d) Invoke Rule Function e) Rule Service Provider Configuration

AGENDA
Introduction to CEP & Business Events Business Events Palette Overview Channels & Destinations Events Concepts Rules & Rule Sets Scorecards

CHANNELS & DESTINATIONS

CHANNELS represent physical connections to a resource

DESTINATIONS

CHANNEL
MESSAGES

DESTINATIONS in a channel represent listeners to MESSAGES from that resource

ARCHITECTURE : CHANNELS & DESTINATIONS

Channel
Subject: orders

Destination orders
map to

Default Event

new_order

TIBCO EMS

Deserialization
Subject: credit

Default Destination credit Serialization

Event

credit_timeout

TYPES OF CHANNELS

TIBCO Rendezvous Channels

Connects BE to RV sources & sinks.

JMS Channels

Connects BE to EMS sources & sinks.

Local Channels

Connects multiple rule sessions at runtime.

AGENDA
Introduction to CEP & Business Events Business Events Palette Overview Channels & Destinations Events Concepts Rules & Rule Sets Scorecards Example

WHAT IS AN EVENT?
ACTIVITY that happens. EXAMPLE: Kicking a ball.

Instance

In Complex Event processing the term Event is overloaded. It means the definition of object that represents the activity (Event type) and also an Instance of that event type.

WHAT HAPPENS WHEN AN EVENT INSTANCE IS CREATED?

1) Memory Assertion Events that are created from incoming messages, are automatically asserted into working memory. At runtime, event instances that are created using rules are not automatically asserted into working memory. You must explicitly assert such events, for example using the Event.assertEvent() function.
Automatic

Business Events
External Environment

Working Memory Create Event instance

Explicit

WHAT HAPPENS WHEN AN EVENT INSTANCE IS CREATED?

2) Memory Acknowledgement Depending upon the object management type BusinessEvents acknowledges the message. Some messages do not require acknowledgement. e.g.: Reliable Rendezvous messages.

TYPES OF EVENTS

SIMPLE EVENT
Life of an Event Time To Live (TTL)

Event Expired!

Zero Positive Negative

After completion of first RTC Cycle. After specified time period has elapsed. Does not expired. Must be explicitly consumed.

Expiry Action -Any Action(s) possible to define in Rule language, to take when a simple event expires. -Expiry actions can be inherited from events parent. -If an event is explicitly consumed in the rule, BusinessEvents does not execute the expiry action.

SIMPLE EVENT
Event Inheritance/Inherits from
Parent Event can NOT have payload.

Can NOT have distinct properties with same name.

Expiry Actions set in child Event overrides the parent event Expiry Actions.

SIMPLE EVENT- HOW THE EVENT DATA IS CARRIED?

Properties has data types. e.g.: String,int,boolean

Payload has complex data structures. e.g.: XML Schema

Extended Properties are used internally & it is reserved for future release.

TYPES OF EVENTS

TIME EVENTS
TIMERS, that are used to trigger rules.

Repeat Every -Creates a new time event after every specified time interval. -Creates specified number of events at each time interval.

Rule Based
-ScheduleTimeEventName() function is used

inside Rule, to create a new time event.

TYPES OF EVENTS

ADVISORY EVENTS
Advisory Events are asserted into the memory when certain conditions occur. It has attributes for category, type & message. Exception
Category: Exception Type: Exception class name Message: Message in class.

Engine Activated Advisory Events

Category: Engine Type: engine.primary.activated Message: Engine <EngineName> activated.

BusinessEvents-ActiveMatrix BusinessWorks Integration

Category: Engine Type: INVOKE BW PROCESS Message: Error message from failed BW Process.

AGENDA
Introduction to CEP & Business Events Business Events Palette Overview Channels & Destinations Events Concepts Rules & Rule Sets Scorecards

Concepts are descriptive entities similar to the object-oriented concepts of a class.

Department
Name Code Manager Employee Event
Code Manager Employee

CONCEPTS

Working Memory
Depar tment Car Pen

-Concepts are automatically asserted into working memory when created, except when concepts are returned by Database query & in the context of in-process integration.
Insert.createInstance() Insert.deleteInstance()

-Concepts needs to be explicitly deleted.

CONCEPTS
Historical Values for a concept property are kept in a ring buffer.

CONCEPTS
History Policy
000362510100

CONCEPT RELATIONSHIP

CONCEPT RELATIONSHIP
Refers
Wheel
Diameter Vendor Rim Type Material

Customer
CustomerID OrderID OrderDetails

Car
Color Vendor Make Model Engine Wheel:Contains

Inherits
Bike Wheel
----- -----------

Car Car Wheel

Contains
Car Wheel
---------------

Car

Car Wheel

Motorbike

AGENDA
Introduction to CEP & Business Events Business Events Palette Overview Channels & Destinations Events Concepts Rules & Rule Sets Scorecards

RULES
A Rule includes a declaration of entity types, one or more separate conditions , which evaluate to true or false, and an action, which is eligible to execute only when ALL the conditions evaluate to true

RULE

Declaration of Entity Types

+ Conditions +

Actions

Rule Priority

Use priorities prudently

A TYPICAL RULE

RULE SETS

A rule set is a container for rules. All rules exist within a rule set. Grouping rules into rule sets enables you to deploy a selection of rule sets in a Business Events Archive (BAR).

EXAMPLE : As seen in the picture, different colored pebbles are contained by different mesh bags. So mesh bag becomes the rule set that contains pebbles which are the rules.

RULE FUNCTIONS

Function for a use at a project level. Not contained in a rule set.

Event Preprocessor Functions

Start Up Functions

Mainly 4 functions

types

of

rule

Virtual Functions

Shut Down Functions

AGENDA
Introduction to CEP & Business Events Business Events Palette Overview Channels & Destinations Events Concepts Rules & Rule Sets Scorecards

SCORECARDS

Serves as a static variable which is available throughout the project. Unlike concepts and events, each scorecard resource is itself a single instance. Use a scorecard resource to track Key Performance Indicators (KPI) or any other information.

AGENDA
State Modeler Database Concepts Out & In Process Integration

STATE MODELER
UML-compliant application.
Initiation

Used to model the life cycle of an instance of concept.

To model the life cycle a state machine resource is used. Within a state machine resource you can configure the states and transitions. Example:Project Life Cycle. Classify Orders.
Closure

Project Life cycle

Execution

Defines transition from state to state based on rules that apply.

Planning

STATE MODEL
Each State model begins with start state and ends with one or more end states . Between these states may be simple, composite and concurrent states connected by transitions.

STATE MACHINE RESOURCE

Exists Within

one

At most one

Instance of Main state Machine

State calls Machine

Call State machine

MODELING ORDER PROCESS

Accounting performs credit check

Customer passes Credit check

Customer places Order

Warehous e checks Inventory Customer receives the Item

Warehous e finds the item in stock

Item sent for Shipment

MAIN STATE MACHINE

Parent State Machine

CALLS CHILD STATE MACHINE

State Machine

Child State Machine

Recursive calls to state Machines not Allowed!

STATE MACHINE STATES

STATES

START
Entry Action

Exit Action

END SIMPLE
Exit Action

Entry Action

COMPOSITE STATE
Composite States are like nested folders. Composite States can contain simple states, other composite states and concurrent states. Credit Check (Composite State) Fulfillment Process

COMPOSITE STATE
Composite States are like nested folders. Composite States can contain simple states, other composite states and concurrent states. Credit Check (Composite State) Fulfillment Process

FAILS Composite State

State1

State2 FAILED

CONCURRENT STATE
Allows multiple state flows to operate at one time. A state machine cannot exit a concurrent state until all its region have finished processing. Can contain composite , simple states.

Multiple processing lanes

AGENDA
State Modeler Database Concepts Out & In Process Integration

DATABASE CONCEPTS
Database Concepts enable you to manipulate the database using a rule or rule function. Business Events
Tables
Insert Update Delete Query

DB Import Utility

Views

DB Concept

Object to Relational Mapping

AGENDA
State Modeler Database Concepts Out & In Process Integration

OUT-OF-PROCESS ACTIVE MATRIX BUSINESS WORKS INTEGRATION

Enables you to send and receive Business Events events in an Active Matrix Business Works engine. Both engines run in separate JVMs.

Business Events Engine

JMS/RV channel

Business Works Engine

Enterprise and External Resources

OUT-OF-PROCESS ACTIVE MATRIX BUSINESS WORKS INTEGRATION ACTIVITIES

IN-PROCESS ACTIVE MATRIX BUSINESS WORKS INTEGRATION

Enables you to integrate ActiveMatrix BusinessWorks and Business Events functionality in one JVM. Enables each product to take advantage of the strengths of the other product. ActiveMatrix BusinessWorks can use Business Events as a light-weight rules engine. BusinessEvents can use transports available in ActiveMatrix BusinessWorks.

JVM
ActiveMatri Business x Business Events Works

 All input and output is done through Business Works.

The BusinessEvents engine cannot communicate with resources outside of the BusinessWorks container.

Less managemen t overhead Simplifies maintenanc e

All input and output is done through Business Events.

The BusinessWorks engine cannot communicate with resources outside of the BusinessEvents container.

BUSINESS WORKS FEATURES

Passes concept/even t /object Business Works process Uses Invoke Rule Function Activity Rule service Provider Configuration (Required only when BW container) To call Business Events Rule Function

Used to identify the location of Business Events application at runtime

BUSINESS EVENTS FUNCTIONS

BusinessWorks.startProcess() BusinessWorks.invokeProcess() -- Invokes a BusinessWorks process in -- Invokes a BusinessWorks process in asynchronous mode and returns the process ID. synchronous mode. -- Upon completion, the BusinessWorks process -- Waits for completion of the process before passes an event to the rule function specified in an returning to the rule or rule function. argument of -- Starts the process engine startProcess(). BusinessWorks.init -- Starts the process engine if it is not already started. -- Returns an event, or null. if it is not already started. () (Optional)

BusinessWorks.cancelProcess() -- Cancels the specified BusinessWorks process. -- Useful for canceling a long running job .

-- Used with BE container only. -- Initializes BW Engine.

BusinessWorks.shutdown() (Optional)

--Shuts down the

BusinessWorks process engine.

DESIGN CONSIDERATIONS RELATED TO CONTAINER

AGENDA
Introduction to Object Management And Fault Tolerance In Memory Object Management Persistence Object Management Cache Object Management

OBJECT MANAGEMENT

OM is used to manage the state of Ontology objects (concepts, state machines, scorecards, simple events and time events) that are created and used by each Rete network.
Three main Options for Object Management

In Memory
Persistence Cache

FAULT TOLERANCE
Fault tolerance mechanism differs with each of the OM options.

In Memory FT is provided at the engine level. Configuration uses various engine properties to define primary and secondary engines.

Persistence Fault tolerance requires a custom solution. Cache FT is provided at the inference agent level. With multi-engine features enabled, fault tolerance and load balancing are provided by the same set of features.

OVERVIEW
Default option

Persistence of Objects : Enables objects to be available for reuse,

either in memory caches or in databases.

Data Recovery : Ability to survive failures.

Partitioned : The ability to partition the objects among multiple JVMs.

Clustered : The ability to maintain multiple copies of each object in different nodes such that if one node fails, another node can take over.

AGENDA
Introduction to Object Management In Memory Object Management Persistence Object Management Cache Object Management

IN MEMORY OBJECT MANAGEMENT

Objects are managed in memory

Rete n/w

Rete n/w

Primar y server

Fault Tolerance Cluster of

two engines using in memory OM

Total System Failure

FT Seconda ry server

FT

NO Recovery Possible maintained

FT Cache Object state is not

FAULT TOLERANCE FOR IN MEMORY OM

BusinessEvents offers priority-based fault tolerance to provide high availability of the BusinessEvents engine process. Fault tolerance provides transitioning between inactive and active states. If two servers have the same priority setting, then the server that joined the group first takes priority in determining the failover and failback order.

FAILOVER BEHAVIOR(IN MEMORY OM FT)

When a node fails, the node with the next highest priority assumes responsibility for that nodes work.

Rete n/w

Rete n/w

Fault Tolerance Cluster of

two engines using in memory OM

Primar y server

FT

Primary Seconda Server ry server

FT

FT Cache

FAILBACK BEHAVIOR(IN MEMORY OM FT)

When a node restarts , it assumes responsibility from the node with the next lowest priority.

Rete n/w Last Primary server

Rete n/w

FT

Primary Seconda Server ry server

FT

Fault Tolerance Cluster of

two engines using in memory OM

FT Cache

STEPS TO CONFIGURE FT IN IN MEMORY OM

Open the BE_HOME\bin\be-engine.tra file in text editor. Configure following properties with same values for all engines. Engine.FT.UseFT true Engine.FT.GroupName group name In each engine property file provide the unique engine name be.ft.nodename

Configure the weight properties to define priorities among servers. Secondary Servers will have lowest priorities. Engine.FT.Weight integer

AGENDA
Introduction to Object Management Memory Object Management Persistence Object Management Cache Object Management

PERSISTENCE OBJECT MANAGEMENT

The PERSISTENCE object management option persists a snapshot of the working memory for each inference agent in the deployed system. Data for each inference agent is persisted to a data store at specified intervals.

PERSISTENCE OBJECT MANAGEMENT CACHE

Small cache for each inference agent ensures that currently used objects are available in memory for improved performance.

YOU CAN CONTROL THE SIZE OF THE CACHE

PERSISTENCE OBJECT MANAGEMENT WORKING

Provides data recovery in the case of a complete system failure. When system comes up, BE restores the working memory to the last check point state.

Receives all of the previously unacknowledged messages.

Data in memory at the time of failure and not yet written on this disk is LOST.

PERSISTENCE OBJECT MANAGEMENT

Affects performance due to disk writes required. Fault Tolerance features are not provided by BusinessEvents. Parameters such as checkpoint interval and property cache size helps us to tune performance. Can also determine how many objects to keep in the data cache, in order to manage JVM memory usage for the application for better performance.

PERSISTENCE OBJECT MANAGEMENT PROPERTIES

A checkpoint is the point in time at which the working memory data is written to disk. Database operations include time, creations, updates, and deletions. The checkpoint interval is theobject in seconds, between writes to disk. Outstanding database operation is one that is held in working memory only. It has not yet been written to the disk. Defines the maximum number of concept ops exceeds that of the numberJVM memorycheckpoint occurs When the number of outstanding DB properties that are kept in the specified, a for this rule session. When the persistence layer performs cleanup, the least recently used(LRU) properties are moved to the When objects are retracted(deleted) from the memory, they are marked with a retraction flag. persistence store, to reduce the number of properties in memory to the specified number. You can delete these objects from the persistence DB, or you can leave them in the DB. Recommended that you delete these retracted objects to avoid accumulating large numbers of Uncheck this field toobjects in DB, but you can keep them for from mining purposes. shutdown. retracted use the persistence database to recover data unplanned system Performance INCREASES, but data is lost in event of SYSTEM FAILURE. DEFAULT PATHS where PERSISTENCE FILES are stored BE_HOME/db/session_name : TIBCO Administrator working_directory/db/session_name : BE Engine

USING CHECKPOINT INTERVAL & OUTSTANDING DB OPS

OUTSTANDING DATABASE OPERATIONS

DATABASE OPERATIONS

AGENDA
Introduction to Object Management Memory Object Management persistence Object Management Cache Object Management

CACHE OBJECT MANAGEMENT

Object Management refers to managing the state of ontology object instances created by each inference agent.
- Manage data objects - Handles recovery

Cache Server abcde

Inference Agent

Rete n/w

CACHE OBJECT MANAGEMENT

Distributed Cache

Cache Server a 1 b c

Cache Server 2 d e

Inference Agent

Rete n/w

CACHE OBJECT MANAGEMENT

Cache Cluster with Load Balancing
requires point to point communication!
Cache Cluster

Cache Server a 1 b c

Cache Server 2 d e

Event z

Inference Agent 1

Inference Agent 2

Rete n/w

Inference Agent 3

Rete n/w

Rete n/w

CACHE OBJECT MANAGEMENT

Cache Cluster with Fault Tolerance
Cache Cluster

Cache Server a 1 b c

Cache Server 2 d e

Event z

Inference Agent 1

Inference Agent 2

Rete n/w

Inference Agent 3

Rete n/w

Rete n/w

CACHE OBJECT MANAGEMENT

Cache Cluster with Fault Tolerance
Cache Cluster

Cache Server a 1 b c

Cache Server 2 d e

Event z

Inference Agent 1

Inference Agent 2

Inference Agent 3

Rete n/w

Rete n/w

AGENDA
Understanding Run to Completion (RTC) Cycle & Conflict Resolution Startup & Shutdown Rule Functions Event Preprocessor Rule functions Virtual Rule Functions Rule Analyzer & Debugger Rule Profiler

UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE

External action changes working memory BusinessEvents executes the first rule action on the agenda and removes it from the agenda

RTC ends

Is the agenda empty ?

BusinessEvents refreshes the agenda

Does the action change working memory ?

UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE

Begins when an external action causes changes to working memory One RTC cycle ends when there are no more rule actions to execute as a result of that initial change. This is also known as forward chaining, or inferencing. During one RTC no new external actions can affect the working memory. An RTC is composed of one or more conflict resolution cycles.

UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE

External action changes working memory

UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE

Working Memory changes

When a message arrives at a destination, the working memory changes. First conflict resolution cycle begins.

Business Events builds the agenda

Business Events examines all rules that are newly true because of the change to working memory and compares them with rule dependencies. The agendas entries are ordered according to rule priority and other criteria.

UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE

External action changes working memory BusinessEvents executes the first rule action on the agenda and removes it from the agenda

BusinessEvents refreshes the agenda

Does the action change working memory ?

UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE

Business Events executes the first rule on the agenda and removes it from the agenda

The rule action does not change working memory and Business Events executes the next rule entry in the agenda (if there is one).

OR

The rule action does change working memory and Business Events refreshes the rule action agenda to account for the changes.

UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE

Business Events refreshes the agenda

Rules that have become newly true are added to the agenda.

OR

Rules that have become false are dropped from the agenda.

OR

Rules that were newly true at the last conflict resolution cycle and are still true remain in the agenda.

UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE

External action changes working memory BusinessEvents executes the first rule action on the agenda and removes it from the agenda

RTC ends

Is the agenda empty ?

BusinessEvents refreshes the agenda

Does the action change working memory ?

UNDERSTANDING RUN TO COMPLETION (RTC) CYCLE

An empty agenda ends the RTC

At some point, no more actions remain to be executed. The conflict resolution has run to completion.

At the end of one RTC, the following happens

Events are sent to destinations. Cache OM: Changes are saved to the cache and written to the backing store. Cache OM, cache-only cache mode: At the end of the RTC, all cache-only objects are removed from working memory. Persistence OM: One transaction is completed and saves changes (enabling rollback in case of failures). Profiler: profiler data is updated.

HOW A RULE BECOMES NEWLY TRUE

A rule is newly true if it has become true due to a change in working memory. A rule that was already true can also become newly true. A rule remains newly true until it is executed or it is removed from the agenda, or the RTC ends.

Runs.scored = 90 NEWLY TRUE

Runs.scored = 95 NEWLY TRUE

ALREADY TRUE

HOW CONFLICT RESOLUTION USES RULE DEPENDENCIES

Before any data enters into the system, BE builds a Rete Network, which has all the rule dependencies, using the rule conditions. All dependencies in a rule are called its dependency set, which is the only thing needed to determine the truth of all the conditions.

customer.name == James;

person.name == John; hasAllAccess(customer);

DEPENDENCY SET

TESTING THE TRUTH OF A RULES CONDITION USING DEPENDENCY SET

Business Events tests each rules dependency set against new set of facts. If facts match the rule dependencies, the rule action is added to the rule action agenda

ORDER OF EVALUATION OF RULE CONDITIONS

Conditions that only involve one scope element
1

FILTERS

Conditions that compare two expressions using == or !=

EQUIVALENT JOIN CONDITIONS

NON EQUIVALENT JOIN CONDITIONS

Conditions involving two or more scope elements other than equivalent joins
3

AGENDA
Understanding Run to Completion (RTC) Cycle & Conflict Resolution Startup & Shutdown Rule Functions Event Preprocessor Rule functions Virtual Rule Functions Rule Analyzer & Debugger Rule Profiler

STARTUP & SHUTDOWN RULE FUNCTIONS

Configured to execute during normal system startup and shutdown. Used to initialize the system & perform more expensive operations so that the system is more efficient at runtime.

TAKE NO ARGUMENT

VALIDITY ACTION

STARTUP & SHUTDOWN RULE FUNCTIONS

Configure these functions in the BUSINESS EVENTS ARCHIVE