Erlang Data Storage Modules

Tutorial
by Flavio Ishii
 Erlang Data Storage Modules

•Modules:
•Erlang Terms Storage (ETS)
•Disk Erlang Term Storage (DETS)
•Mnesia Distributed DBMS
• Data Storage Design

Flavio Ishii, Sept. 17, 2009

 Tuples

•A Tuple groups items into one entity/

term.

•Used in Erlang data storage modules.

Flavio Ishii, Sept. 17, 2009

 Tuples
> N = { “Flavio Ishii” }.

> S = [“cycling”,"ultimate frisbee" ].

> P = { person, { 1, N, S } }.

> { person, Properties } = P.

> Properties.

> { Id, Fullname, Sports } = Properties.

> X = lists:append( Sports, [“basketball”] ).

> X.

Flavio Ishii, Sept. 17, 2009

 ETS
• Erlang Term Storage
• Constant access time (excl. ordered_set)
• Table Options:
• access rights: private | public | *protected
• types: *set | ordered_set | bag | duplicate_bag
• transfer ownership: {heir, Pid, HeirData}, or
give_away/3
• Various methods to query the table: mnesia
function (pattern matching) or qlc

* default
Flavio Ishii, Sept. 17, 2009
 ETS Table Access Rights
• private - Only owner process can read/write

• public - All processes with table id can read/write

• protected - Only owner process can write and

any process with table id can read.

Flavio Ishii, Sept. 17, 2009

 ETS Table Types
• set - unique {key, value}, inserts may overwrite.
• [ {b,2}, {a,5} ]
• ordered_set - key ordered and unique tuple
• [ {a,5}, {b,2} ]
• bag - duplicate key allowed
• [ {a,5}, {b,2}, {b,4} ]
• duplicate_bag - duplicate tuple allowed
• [ {a,5}, {b,2}, {b,2} ]
• hash tables: set, bag, and duplicate_bag
• balanced binary tree: ordered_set
Flavio Ishii, Sept. 17, 2009
 ETS
> Tab = ets:new(user, []).

> ets:insert(Tab, [{1,flavio},{2,ralph},{3,melissa},{4,bob}]).

> ets:info(Tab).

> ets:match(Tab,'$1').

> ets:match(Tab,{'$1',bob}).

> ets:match_object(Tab,{'$1',bob}).

> ets:select(Tab,[{{'$1',bob},[],['$$']}]).

> ets:select(Tab,[{{'$1',bob},[],['$_']}]).

> qlc:eval( qlc:q([{Y} || {X,Y} <- ets:table(Tab), (X > 2) and

(X < 4)])).

Flavio Ishii, Sept. 17, 2009

 DETS
• Stores persistent data in disk/file (disk seeks)
• Process needs to open the file.
• Table may be shared by local processes.
• No ordered_set table type.
• Must be properly closed when process is
terminated.
• Checks for consistency on startup after crash.

Flavio Ishii, Sept. 17, 2009

 DETS
> dets:open_file(user, []).

> dets:insert(user, [{1,flavio},{2,ralph},{3,melissa},

{4,bob}]).

> dets:info(user).

> dets:match(user,'$1').

> dets:match(user,{'$1',bob}).

> dets:match_object(user,{'$1',bob}).

> dets:select(user,[{{'$1',bob},[],['$$']}]).

> dets:select(user,[{{'$1',bob},[],['$_']}]).

> qlc:eval( qlc:q([{Y} || {X,Y} <- dets:table(user), (X > 2)

and (X < 4)])).

Flavio Ishii, Sept. 17, 2009

 Mnesia
• Erlang’s distributed DBMS
• ETS & DETS core
• Location Transparency
• Fault Tolerance via Table Replication &/or
Fragmentation across nodes
• Transactions, Locking, & Dirty Operations
• Schema Manipulation at runtime
• ACID Properties

Flavio Ishii, Sept. 17, 2009

 Mnesia ACID Properties
• Atomicity - succeed on all or no nodes
• Consistency - ensure consistent state
after crash
• Isolation - isolate manipulations on
same record
• Durability - changes are committed to
disk.

Flavio Ishii, Sept. 17, 2009

 Mnesia Table Options
• {type, Type} % set, ordered_set, bag
• {record_name, Name}
• {ram_copies, NodeList} % fastest
• {disc_copies, NodeList} % RAM and disc copies
• {disc_only_copies, NodeList} % slowest
• {attributes, AtomList}
• {index, IndexAtomList}

Flavio Ishii, Sept. 17, 2009

 Terminal Time!
• Create record structure
• Create schema and tables
• Insert records
• Query records

Flavio Ishii, Sept. 17, 2009

 api.hrl
% this file defines the records

-record(counter,{id_name,value}).
-record(user_details,
{password,firstname,lastname,sports=[]}).
-record(user,{id,username,user_details}).

Flavio Ishii, Sept. 17, 2009

 Part 1/4 of api_db.erl
% this file defines the records
-module(api_db).
-export([init_db/0, add_sport/2, mne_fun_query/1, qlc_query/1]).
-include("api.hrl").
-include_lib("stdlib/include/qlc.hrl").

init_db() ->
mnesia:create_schema([node()]),
io:format("Mnesia schema created~n"),

mnesia:start(),
mnesia:change_table_copy_type(schema, node(), disc_copies),
mnesia:create_table(counter, [{disc_copies,[node()]}
, {attributes, record_info(fields,
counter)}]),
mnesia:create_table(user, [{disc_copies,[node()]}, {index, [username]}
, {attributes, record_info(fields, user)}]),

io:format("Mnesia tables created~n"),

add_sample_data().
Flavio Ishii, Sept. 17, 2009
 Part 2/4 of api_db.erl
insert_user(UserRecord) ->
case UserRecord#user.id =:= undefined of
true -> NewUser = UserRecord#user{ id =
mnesia:dirty_update_counter(counter, user_id, 1) };
false -> NewUser = UserRecord
end,
% mnesia:dirty_write(NewUser). or use a transaction...
Fun = fun() ->
mnesia:write(NewUser)
end,
mnesia:transaction(Fun).

add_sample_data() ->
Flavio = #user{username="flavio"
, user_details=#user_details{ password="mypassword", firstname="Flavio",
lastname="Ishii", sports=["biking","basketball"]}},
insert_user(Flavio),
Bob = #user{username="bob",
user_details=#user_details{ password="hispassword", firstname="Bob",
lastname="The Builder", sports=["soccer"]}},
insert_user(Bob),
io:format("Users added to table.~n").

Flavio Ishii, Sept. 17, 2009

 Part 3/4 of api_db.erl
% > api_db:add_sport("flavio","football").
add_sport(Un,NewSport) ->
[User] = mne_fun_query({username,Un}),
UserDetails = User#user.user_details,
Sports = UserDetails#user_details.sports,
NewList = lists:append(Sports,[NewSport]),
NewUser = User#user{user_details=#user_details{sports=NewList}},
insert_user(NewUser).

% This is one method of querying a user.

% > api_db:qlc_query({username,"flavio"}).
qlc_query({username,Username}) ->
F = fun() ->
qlc:e(qlc:q([U#user.user_details || U <- mnesia:table(user)
, U#user.username =:= Username
]))
end,
mnesia:transaction(F).

Flavio Ishii, Sept. 17, 2009

 Part 4/4 of api_db.erl
% This may not be ideal but it demonstrates the use of Pattern Matching
% > api_db:mne_fun_query({username,"flavio"}).
mne_fun_query({username,Un}) ->
MatchHead = #user{username='$1', _='_'},
Guard = [{'=:=','$1',Un}],
Result = ['$_'],
MatchSpec = [{MatchHead, Guard, Result}],
mnesia:dirty_select(user, MatchSpec);

% > api_db:mne_fun_query({sport,"soccer"}).
mne_fun_query({sport,Sport}) ->
UserDetails = #user_details{_='_',sports='$1'},
MatchHead = #user{user_details=UserDetails,_='_'},
Guard = [{'=:=','$1',[Sport]}],
Result = ['$_'],
MatchSpec = [{MatchHead, Guard, Result}],
F = fun() -> mnesia:select(user, MatchSpec) end,
mnesia:transaction(F);

mne_fun_query(_) -> io:format("No match~n").

Flavio Ishii, Sept. 17, 2009

 Head to Head Feature List

ETS DETS Mnesia

Persistent storage X X
Complex search queries X X X
Distributed Replicated data storage X X X
Table Fragmentation X
Fault tolerance via replication X X X
Tight Erlang coupling X X X
Complex objects & relationships X
Dynamic reconfiguration X X X
Table indexing X
Distributed Transactions X

Flavio Ishii, Sept. 17, 2009

Possible Design Permutations

Flavio Ishii, Sept. 17, 2009

More Permutations

• Table access (i.e. public,

protected)

• Table type (i.e. set, bag...)

• Table locks
• Distribution via replication
and/or table fragmentation

Flavio Ishii, Sept. 17, 2009

 Useful Mnesia Functions
• change_table_access_mode/2
• change_table_copy_type/3
• backup/2
• install_fallback/2
• restore/2
• dump_tables/1
• lock/2
• move_table_copy/3
• async_dirty/2
• sync_dirty/2
• add_table_copy/3
• transform_table/3
• change_table_frag/2
• activity/4
Flavio Ishii, Sept. 17, 2009
 Resources
• Programming Erlang, Joe Armstrong
• ets, dets, qlc, mnesia manuals:
• http://www.erlang.org/doc/man/
• Mnesia User’s Guide:
• http://www.erlang.org/doc/apps/mnesia
• Mnesia - A Distributed Robust DBMS for
Telecommunications Applications, Håkan
Mattsson, Hans Nilsson and Claes Wikström

Flavio Ishii, Sept. 17, 2009

 Contact Info

USASK - MADMUC Lab

http://flavioishii.com
@flavioishii

Flavio Ishii, Sept. 17, 2009