Prof.

Hasso Plattner

A Course in
In-Memory Data Management
The Inner Mechanics
of In-Memory Databases

October 8, 2013

This learning material is part of the reading material for Prof.

Plattners online lecture "In-Memory Data Management" taking place at
www.openHPI.de. If you have any questions or remarks regarding the
online lecture or the reading material, please give us a note at openhpi-
imdb@hpi.uni-potsdam.de. We are glad to further improve the material.
Chapter 34
Bypass Solution

As illustrated throughout the course, in-memory data management can en-

able significant advantages to data processing within enterprises. However,
the transition for enterprise applications to an in-memory database will re-
quire radical changes to data organization and processing, resulting in major
adaptations throughout the entire stack of enterprise applications. By consid-
ering conservative upgrade policies used by many ERP system customers,
the adoption of in-memory technology is often delayed, because such rad-
ical changes do not align well with the evolutionary modification schemes
of business-critical customer systems. Consequently, a risk-free approach is
required to help enterprises to immediately leverage in-memory data man-
agement technology without disruption of their existing enterprise systems.
We propose a transition process that allows customers to benefit from
in-memory technology without changing their running systems. This is a
step by step, non-disruptive process that helps to transform traditionally
separated operational and analytical systems into what we believe is the
future for enterprise applications: transactional and analytical workloads
handled by a single, in-memory database.
Within the first step of the transition, an in-memory database will run
in parallel to the traditional database and the data will be stored in both
systems. Secondly, new side-by-side applications using transactional data of
the ERP system can be developed. In the next step, a new Business Intelli-
gence (BI) solution can be introduced that is able to answer flexible, ad-hoc
queries and operate without materialized views, aggregating all necessary
information on the fly from the transactional data. Complex ETL processes
are not required any longer. Finally, the traditional disk-based database of
the ERP system can be replaced by a column-oriented dictionary-encoded
in-memory database. This transition is described in more detail in the next
section.

213
214 34 Bypass Solution

34.1 Transition Steps in Detail

Bypass Solution for Transition

First of all, we start with a commonly found initial architecture of an existing
enterprise solution as illustrated in Figure 34.1.
Typically, it consists of multiple OLTP and OLAP systems, each of them
running on separate databases. The OLAP system consolidates data from
multiple OLTP systems and external data sources. A costly and time-
consuming ETL process between the OLTP and OLAP systems is used to
pre-aggregate data for the OLAP system. Based on this architecture, the
non-disruptive transition plan that we call bypass solution has been de-
veloped.

Todays System Traditional BI

ETL OLAP
ERP Engine

Traditional DB
w/ Cubes'
Traditional DB'

Fig. 34.1: Initial Architecture

4'

In the first step of this approach (see Figure 34.2), the IMDB is installed
and connected to the traditional database.
The only dierence will be in data representation: data will be stored in
columns. An initial load to the in-memory database (IMDB) creates a copy
of the existing system state with all business objects in the IMDB. In spite of
the huge volume of data to be reproduced, first experiments with massively
parallel bulk loads of customer data have shown that even for the largest
companies this one-time initialization can be done in only a few hours.
After the initial load, the two storages will be maintained in parallel,
every document and change in a business object is stored in both databases.
For this, established database replication technologies are used. The high
compression rate in a column store helps to decrease the amount of main
memory required for such parallel use of two databases, and hence it does
not lead to a significant waste of resources. At the same time, using the
parallel installation of the IMDB, we can estimate performance and memory
consumption benefits of this architecture for concrete business cases and
prove the need of moving the system to the new data storage.
 STEP 1: Install and run the in-memory database in parallel

34.1 Transition Steps in Detail 215

Todays System with IMDB Traditional BI

ETL OLAP
ERP Engine

Traditional DB
w/ Cubes'
IMDB
Traditional DB'
'

SSD'

Fig. 34.2: Run Data Replication to the IMDB in Parallel

Bypass Solution for Transition

In a second step, new applications can be developed leveraging the po-
tentials of the newSTEP
technology
1: Install(see
and Figure
run the 34.3). Thesedatabase
in-memory applications only read
in parallel
the replicated ERP system data. If they want to write data, they either use the
existing interfaces of the ERP system or, if they want to store additional data,
they use a separate segment in the in-memory database. That way, business
value by new revolutionary applications can be generated from the first day
on.

Todays System with IMDB Traditional BI

ETL OLAP
ERP Engine

Traditional DB
w/ Cubes'
IMDB
Traditional DB'
'

New
Applications
SSD'

Fig. 34.3: Deploy New Applications

216 34 Bypass Solution

In a third step, which can be done in parallel with the previous ones, the
BI system is ported to the IMDB as illustrated in Figure 34.4. This will help to
achieve another gain in reporting performance in comparison to disk-based
OLAP systems.
The dierence to the traditional system is that all materialized data cubes
and aggregates will be removed. Instead, aggregates are computed on the fly
and all data-intensive operations are pushed to the database level. In com-
parison with storing all materialized aggregates and indices, this reduces
the amount of main memory, which is necessary for the OLAP system. It
immediately leads to the following benefits: the data cubes in a traditional
BI system are usually updated on a weekly basis or even less frequently.
However, executives, management, and all other decision makers often de-
mand up-to-date information. By running OLTP and OLAP systems on the
same IMDB platform, this information can be delivered in real time. Another
advantage is that the ETL process is radically simplified as the complex cal-
culation of aggregates is omitted. The ETL process can partly be replaced by
the same replication mechanism used between the traditional ERP database
and the parallel installation of the in-memory database. Therefore, we called
the replication in Figure 34.4 simply EL, since no transformation takes
place any longer, just extraction and load remain. Furthermore, the Business
Intelligence (BI) -system will be more flexible as complex cube management
and maintenance operations are abandoned and complexity is reduced.
In most cases, the migration to the in-memory database BI system can be
conducted automatically. This is relatively simple as existing materialized
views are replaced by non-materialized views. Analytical queries can be
rewritten by generators. However, in complex migrations scenarios, factors
such as dierent SQL dialects, data structures, or software parts that rely
on additional data in proprietary formats may pose an obstacle and might
require manual interventions.
The final step of the suggested solution can be executed when the cus-
tomer is comfortable with the parallel in-memory solution. In this step, the
traditional OLTP database is switched o. After that, the customer works
only with the consolidated in-memory enterprise system that is used for
both transactional and analytical queries (see Figure 34.5).
The BI system could theoretically be replaced in a setup with only one
ERP system. Reality shows that many OLTP systems exist and external data
has to be integrated into the analytical system. To this end, the traditional
BI system is used as a data platform consolidating data from these OLTP
systems and external sources.
Additionally, during system evolution, new extensions to the data model
are possible. Adding new tables and new attributes to existing tables in the
column store is done on the fly and this speeds up release cycles significantly.
 Figures for Book: Step 3

34.2 Bypass Solution: Conclusion 217

Todays System with IMDB Traditional BI with IMDB

EL OLAP
ERP Engine

Figures for Book: Step 4 IMDB

w/o Cubes'
IMDB
Traditional DB'
'

New
Applications
SSD'

Fig. 34.4: Traditional BI system on IMDB

ERP Future EL OLAP

Releases Engine

OLTP & OLAP

IMDB
w/o Cubes'
IMDB
'

New
Applications
SSD'

Fig. 34.5: Run OLTP and OLAP on IMDB

34.2 Bypass Solution: Conclusion

As discussed above, the suggested bypass solution introduces a risk-free and

non-disruptive transition to the in-memory database technology. The impor-
tance of this transition can be backed-up by selected customer examples: For
a large financial service provider, the analysis of 33 million customer records
could be reduced from 45 minutes on the traditional DBMS to 5 seconds
on an IMDB. This increase in speed fundamentally changes the companys
opportunities for customer relationship management, promotion planning,
and cross selling.
218 34 Bypass Solution

In a similar use case, a large vendor in the construction industry is using

an IMDB to analyze its nine million customer records and to create contact
listings for specific regions, sales organizations, and branches. Customer
contact listing is currently an IT process that may take two to three days
to complete. A request must be sent to the IT department who must plan a
background job that may take 30 minutes and the results have to be manually
returned to the requester. With an IMDB, sales people can directly query the
live system and create customer listings in any format they wish, in less than
10 seconds.
Concluding, the use of in memory technology can lead to a qualitative
change in the business processes of an enterprise. The transaction that took
days using the traditional process, can now be performed in the foreground
on the fly. That will change the way of thinking and optimize many business
processes in enterprises.