Business Activity Monitoring (BAM) with Complex Business Event Processing (BEP/CEP)

Business Activity Monitoring (BAM) has been for a while. However, it has been limited to real-time dashboard applications. Basically, it displays visual real-time graphs and charts based on KPIs or Key Performance Indicators. It lacks Complex Event Processing (CEP) capabilities that businesses require. Dashboards are just a tiny segment of BAM. Current BAM cannot meet the business requirements. For example, dashboards show only snapshots of information. It does not indicate what transitions have happened. In addition, managers do not keep watching them all the time. Vital events can be missed. Obviously, better framework is BAM with CEP.

Complex Business Activity Monitoring with Event Processing (CAMEP) combines the two as a single frame work. Rosella Analytic Platform is an integrated end-to-end developer environment which provides all the needed ingredients for CAMEP. It incorporates the followings;

  • BI: Statistics, KPIs, DW/DM, GIS, reporting, charting,...
  • Expert systems: User-friendly SQL-based RME-RETE rule specification language.
  • Predictive modeling: Neural network, decision tree, regression, time-series analysis, ...
  • Persistent fact management: Fast, full ACID transaction support, Real-time cross-platform data replication, ...
  • Event handling: Extensible event handlers can meet any business requirements, ...

The following figure shows the architecture of CAMEP of Rosella Analytic Server;

CEP BAM Architecture.

BAM feeders collect information from operational databases, data warehouses, data marts, and from other sources, and feed it into RME-RETE expert systems. RME-RETE engine evaluates rules and invokes event handlers to generate business events. Rules are developed by business users and analysts.

RME-RETE – Expert Systems Engine

The core of CAMEP is the RME-RETE engine. Rete engine is a de facto industry standard technique for rule-based expert systems. It is also known as expert system shells. The problem with conventional implementations of Rete engines is that they are based on pattern-matching algorithms which are not only un-intuitive but also difficult to learn, especially, by non-IT users such as business analysts, medical doctors, and domain experts. As a consequence, rules have to be extracted from domain experts and then translated by IT knowledge engineers. This process not only takes time to develop but also makes it difficult to extract domain knowledge. The worst part is that detection of errors can be extremely difficult since domain experts cannot verify what the written rules exactly mean.

To alleviate this problem, RME-RETE (Rule-based Model Evaluation - Rete) has been developed to use SQL-like language which can be learnt by business users and domain experts very easily and quickly. As a matter of facts, users already familiar with SQL can learn it within an hour! Non-IT professional users can write RME-RETE expert system rules by themselves without IT professional help. RME-RETE opens a new era for business users and domain experts.

SQL-like language for Business users!

The popularity of SQL as the preferred database query language has been largely due to the natural intuitiveness of the language as well as the richness of logical expressions. SQL is very easy to learn. Expressions written in SQL is very easy to understand and hence has been widely used by many non-IT professional such as business analysts and users, as well as by IT professionals. RME-RETE is based on SQL-99 syntax supporting most logical and arithmetic language features of the language.

The following shows examples of RME-RETE rules. These rules show how newly developing sales/market trends can be detected automatically and generate notification messages;

// SALES TREND MONITORING;
RULE 1: // if sales dropped over 5% with correlation coefficient 0.3 and over;
IF TIMESERIES(RR, LINEAR, 1, Month1, Month2, Month3, Month4)) >= 0.3 
   AND TIMESERIES(GROWTH_RATE, LINEAR, 1, Month1, Month2, Month3, Month4) < -0.05 
   THEN SET sales.trend AS 'declining' END;

RULE 2: // if sales increased over 15% with correlation coefficient 0.3 and over;
IF TIMESERIES(RR, LINEAR, 1, Month1, Month2, Month3, Month4)) >= 0.3 
   AND TIMESERIES(GROWTH_RATE, LINEAR, 1, Month1, Month2, Month3, Month4) > 0.15 
   THEN SET sales.trend AS 'increasing' END;

RULE 3: // alert if something detected;
IF sales.trend IS NOT NULL 
   THEN THROWEVENT('alert', Region, Channels, sales.trend) END;

The following rule shows how RME-RETE rules can be used in medical diagnosis;

RULE 4:  // GENETIC DIAGNOSIS;
IF patient.gene.k12h2 PLIKE '*-C-*(2,4)-C-*(3)-[LIVMFVWC]-*(8)-H-*(3,5)-H-*'
       AND PREDICT("model1") USING(patient.age, patient.gender, ...) > 0.34
       THEN SET patient.diagnosis AS 'has cancer gene' END;

For more, please read RME-RETE Rule Specification Language.

Event Handling

RME-RETE engine evaluates rules and responds by generating events to event handlers. For example, when there is a violation of regulations, users may write rules that inform to relevant business managers. Event handlers are normally user provided functions. They may include;

  • Send email messages.
  • Send SMS text messages.
  • Log messages into files.
  • Log messages into databases.
  • Call other services.
  • and so on.

These functions can be easily implemented using extensible APIs.

Predictive Modeling

The core concept of RME-RETE is Rule-based Model Evaluation (RME) of predictive models. RME-RETE incorporates predictive models with many logical and mathematical expressions of SQL-99. By incorporating advanced predictive models into Rete engine, RME-RETE provides a superb platform for advanced rule-based expert systems. RME-RETE supports the following predictive models;

  • Neural network: Neural network is a very powerful predictive modeling technique. CMS/RME-RETE provides neural modeling tools which are very robust and intuitive but still very easy to use. Neural network can be used to predict numerical scores as well as classifications.
  • Decision tree: Decision tree is primarily used for classification task. In addition, CMS decision trees can also predict statistical probability of events.
  • Regression: Regression is widely used technique by statisticians. It can predict numerical scores. RME-RETE can include regression models in two ways: as mathematical functions and as a CMS model.
  • Time-series regression and analysis: RME-RETE has built-in time-series regression functions. It can explore various ways and find best regression functions based on a set of RME-RETE facts.

For more, please read Predictive Modeling Software.

High Performance!

RME-RETE uses compile-time analysis to build efficient run-time rule activation and evaluation environment. Rules are activated very efficiently. Facts are also managed and fetched using fast binary search techniques. Note that with binary search techniques, fetching a fact out of 1 million facts takes 20 only comparative steps at the worst case. High performance is the theme of RME-RETE!

Persistence Fact Management - Rosella

RME-RETE engine is coupled with embedded Rosella DBMS as a persistent fact manager. Rosella DBMS is a high performance relational database system. It combines the bests of disk-based and main-memory database systems. Rosella DBMS can use huge main-memory for staging data and process with ultra fast sorting and indexing. RME-RETE persistent fact managers provide;

  • Fast data access using embedded connections and special prepared statements.
  • Big main-memory loaded data (up to 512GB).
  • Real-time cross-platform data replication or real-time data backup over different operating systems.
  • Full ACID transaction support provides logically consistent transactional event processing.
  • Fast indexing provides fast access to and update of facts.
  • Robust recovery ensures that important data updates will not be lost even with disasters.

This translates into fast and reliable real-time business activity monitoring systems!

Applications of BEP BAM

BEP BAM or CAMEP powered by RME-RETE is ideally suited for many mission-critical real-time tasks. The robustness of the rule specification language combined with advanced predictive modeling provides a superb platform for RTE applications. Real-time applications need high processing ratios. Embedded deployment of applications will provide the needed speed, if SOAP/SOA based deployment cannot meet! Applications of RME-RETE may include;

  • Automated data mining.
  • Customer-churn detection.
  • Cross-sell advisor.
  • Algorithmic share trading.
  • Real-time transaction audit.
  • Account auditing.
  • Intrusion detection.
  • Risk management.
  • Fraud detection: insurance, healthcare.
  • Medical diagnostic expert systems.
  • and so on.