MySQL Conference & Expo 2011

MariaDB
Dynamic Columns

Michael “Monty” Widenius monty@askmonty.org

Oleksandr “Sanja” Byelkin sanja@askmonty.org

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

 RDBMS doesn't solve all common problems

The (web) store problem:

All items need: ID, Type, Price, Country, Manufacturer)

A T-Shirt has the following additional properties:

Size, color...
A computer has the following additional properties:
CPU, MHz, memory, Watt...

There is no easy way to store many different types into a

relational database!

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

 RDBMS doesn't solve all common problems

One common solutions to this is:

● Store all the 'extra' columns in a BLOB in some format

(HTML?)
● You need a lot of extra work to manipulate the blob

● Hard to access column data (usually done in client)

● Overhead in storage (especially when you use HTML)

● All values are 'text'

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

 RDBMS doesn't solve all common problems

Another common solution:

● Create one table for all the 'extra' columns:

CREATE TABLE extra

(id int auto_increment, extra_column_id char(10),
value varchar(255));
INSERT INTO items set type=“t-shirt”, price=10;
INSERT INTO extra (NULL, LAST_INSERT_ID(), “color”,
“Blue”),(NULL, LAST_INSERT_ID(), “Size”, “M”);
The problems with this approach is:
● Every access to an extra column requires a key/row

lookup
● Slow performance (if database is not optimized for this)

● Big overhead in storage (especially in index)

● Risk for errors as data is not typed

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

 Dynamic columns

Dynamic columns is a bridge between relational

databases and non relational databases
● With dynamic columns all extra columns are stored in a
packed blob, maintained by the database.
● You can add more columns, remove or query them for a

row.
● You can access columns in the server or retrieve the full

blob to the client and manipulate it there.

● You can use virtual columns to create indexes on some

values.
● True indexes for dynamic columns is planned for later.

● Implemented through functions for use by ODBC, & etc.

● First implementation uses integer to access columns.

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

 Dynamic columns: supported
types
● unsigned int
● int

● char [character set <cs>]

● double

● decimal [(<len>, <frac>)]

● time

● date

● datetime

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

 Dynamic columns: syntax

Creating a table with a dynamic column for the store:

CREATE TABLE item (

ID int auto_increment primary key,
Type_id int,
Price decimal(7,2),
Country_id int,
Manufacturer_id int,
extra blob);

Where column 'extra' is dedicated to store dynamic

columns. It could be any column able carry text.
Notice: MySQL is a registered trademark of Sun Microsystems, Inc.
 Dynamic columns: syntax

Creating/initializing a dynamic_column:

COLUMN_CREATE(column_nr, value [as type],

[column_nr, value [as type]], ...)

INSERT into item values

(NULL, 1 /* T-shirt */, 10, 1 /* Germany */, 1 /* Nike */,
COLUMN_CREATE(1 /* color */, "Blue", 2 /* Size */, "M"));

INSERT into item values

(NULL, 2 /* computer */, 1000, 1 /* Germany */, 2 /* intel
*/,
COLUMN_CREATE(3 /* cpu */, "T9400", 5 /* MHz */, 800 as
unsigned int));

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

 Dynamic columns: syntax

Updating a dynamic column:

COLUMN_ADD(string, column_nr, value [as

type],
column_nr, value [as type]]...)

UPDATE item SET extra=

COLUMN_ADD(extra, 6 /* Memory */, 2048)
WHERE id=2;

If the column already exists, it will be overwritten.

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

 Dynamic columns: syntax

Deleting a dynamic column (if it exists):

COLUMN_DELETE(string, column_nr1,
column_nr2, ...);

UPDATE item SET extra=

COLUMN_DELETE(extra, 6)
WHERE id=2;

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

 Dynamic columns: syntax

Querying a dynamic column:

COLUMN_EXISTS(string, column_nr);
SELECT
ID, Type_id, Price, Country_id, Manufacturer_id from item
where COLUMN_EXISTS(extra, 3);

Querying which columns exist:

COLUMN_LIST(string);
SELECT COLUMN_LIST(extra) FROM item WHERE id=1;
→ “1,2”

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

 Dynamic columns: syntax

Retrieving a dynamic column:

COLUMN_GET(column_nr, string as type);

SELECT id, COLUMN_GET( 1 /* color*/, extra as char)

from item;
→ 1 Blue
→ 2 NULL

You can of course also do things like:

SELECT id, COLUMN_GET(1, extra as char) FROM item
where Type_id=1
order by COLUMN_GET(1, extra as char);

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

 Dynamic columns: C library

C Library allows the same manipulations with dynamic

columns on the client side.

A description can be found in the worklog

http://askmonty.org/worklog/Server-Sprint/?tid=34 and in the
source.

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

 Dynamic columns: encoding

How is the dynamic column encoded?

Header:
<flag><number_of_columns>
Sorted index:
<column_nr1><offset/type><column_nr2><offset/type>
Each column is stored as:
<data1><data2>

Where 'offset' is offset from beginning of data part, 'type' is

3 bits in offset.
Length of the data could be calculated by offsets of 2
neighbor fields.
Notice: MySQL is a registered trademark of Sun Microsystems, Inc.
 Dynamic columns: data encoding

● Unsigned integer is just a variable integer field.

● Signed integer coded to make the variable size efficient:

0 → 0
-1 → 1
1 → 2
-2 → 3
2 → 4
...
● Double, date, time, and date time are fixed-size fields
● String stores collation number and the string

● Decimal stores sizes of parts before and after decimal point

and decimal in MySQL format.

● NULL – means removing the field.

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

 Dynamic columns: avaliability

When will dynamic columns be available?

First version is already pushed in separate tree for testing.

lp:~maria-captains/maria/5.3-mwl34

Should be available in main MariaDB 5.3 within several

weeks.

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

 Dynamic columns: plans

● Adding name directory

● Adding functional indices

● Supporting popular NoSQL data exchange formats

(for example, JSON or XML)

● Adding engine support for NoSQL databases as

HBase

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.

 Thanks

Q&A

Notice: MySQL is a registered trademark of Sun Microsystems, Inc.