Tag Archives: mnesia

Introduction to Mnesia I


Warning: Invalid argument supplied for foreach() in /home/public/wp-content/plugins/ionhighlight/Text/Highlighter/Renderer/Html.php on line 308

Today in my Advanced Concepts in Operating Systems class I led the discussion on the Mnesia paper from PADL’99, while this paper has numerous typos it does do an excellent job highlighting the features and advantages of Mnesia. For those of you who are unaware, Mnesia is a distributed, fault-tolerant object DBMS written in Erlang. One thing about Mnesia that I have found to be lacking is a tutorial written for the lay person from the ground up, this gap I intend to try and fill. This multi-segment tutorial assumes you have knowledge of Erlang, and the basic  concepts of manipulating data with DBMSes, other than that, I hope to provide enough information and code to demystify a fairly complex system. However, I still am on the road to mastery, so if I make any errors, or you have any tips for improvement, I’d be happy to add them in.

To get started, start up the Erlang shell (erl) with a name, I will use -sname foo in the following examples. Below is a transcript of starting up and creating a disk-based

ranok@orion:~/Desktop$ erl -sname foo
Erlang R13B01 (erts-5.7.2) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]
 
Eshell V5.7.2  (abort with ^G)
(foo@orion)1> mnesia:create_schema([node()]).
ok
(foo@orion)2> mnesia:start().
ok

The most important call made here, create_schema, takes a list of nodes to replicate the schema table to on disk. You can add additional disk-based nodes or ram-based copies later (we’ll get to the details later). After you’ve created the schema (this will make a folder for all the Mnesia table data), you can start the application with the start function.

Now that we have the database running, we need at least one table to store the data in, we will start with a very simple record to just store simple key/value data. The nice thing about Mnesia, is that the data we store can be pretty much anything, from a simple atom to a function. We will start learning the basics from the mnesia_test module, which I have uploaded here.

The first few lines of the module start off like any Erlang code, a module declaration, what functions to export, including the QLC (Query List Comprehensions) include file (you may need to find it for your system) and a record definition:

-record(data, {key, value}).

Which will define our record for the table also named data. In the function setup_and_start/0, we tie in what we already went over with the create_table function, which in our case looks like

mnesia:create_table(data, [{disc_copies, [node()]}, {attributes, record_info(fields, data)}])

The create_table function has a number of options, the most basic of which we will deal with at the moment: the name of the new table, where and how the table will be stored and what fields the table has (this code used the record_info() function to pull those out of the data record for us). Now that we have our table, we need a way to get the data in and out of it.

Many databases provide the ability for multiple queries to be joined into one transaction to be executed atomically. Mnesia is no different, but for the most part, all queries are executed through the transaction manager (there is a dirty interface which will be discussed later), this makes working in a distributed environment much easier. The way to perform a transaction in Mnesia is to pass a fun to the mnesia:transaction() function that will atomically run.

The actual function to enter data (both insert and update) is write(Record). We wrap this into the mnesia_test:insert(Key, Val) function displayed below:

insert(Key, Val) ->
Record = #data{key = Key, value = Val},
F = fun() ->
mnesia:write(Record)
end,
mnesia:transaction(F).

To now retrieve this data back from the database, the read function is now used, the read function takes two arguments: the table name (in this case, data) and the key to retrieve. The mnesia_test retrieve function wraps this nicely for us, and is shown below:

retrieve(Key) ->
F = fun() ->
mnesia:read({data, Key})
end,
{atomic, Data} = mnesia:transaction(F),
Data.

mnesia:transaction will either return {aborted, Reason} or {atomic, Rows}, where Rows is a list of all the retrieved data. If the key we tried to retrieve could not be found, then it will return an empty list.

Say however, we want to search the table for certain values that are not the index of the table. For that there is the matching functions, the simplest of them is the match_object whose usage can be seen here:

search(Val) ->
F = fun() ->
mnesia:match_object(#data{key = '_', value = Val})
end,
{atomic, Data} = mnesia:transaction(F),
Data.

As you can see, simply fill in all the values that are known and that you want to search for, and use the ‘_’ unmatched value for all the other values. This transaction will return the same forms as the read transaction.

There is another method for filtering through Mnesia tables, which is very similar to the list comprehensions builtin to Erlang which is called QLC. The QLC version of the above function is below:


search_qlc(Val) ->
F = fun() ->
qlc:eval(
qlc:q(
[X || X <- mnesia:table(data), X#data.value == Val] )) end, {atomic, Data} = mnesia:transaction(F), Data.[/code] What the query here is doing is is returning a list of Xs where every possible X comes from our data table, and X#data.value == Val. This should be very intuitive for those of you who are familiar with list comprehensions. What qlc:q() does is form a query handle (much like a function object) which gets evaluated by qlc:eval() inside of our transaction object. Again, this will return the same values from mnesia:transaction. Well, that about covers the basics of Mnesia, you now should be able to setup Mnesia on your computer, create a table and insert/retrieve data from it. In the next installment, we will look at distributed Mnesia and the dirty interface, which provides faster queries by bypassing the transaction manager. After that we will put all of what we've learned into creating a system that will take advantage of Mnesia and give a pseudo real-world problem a fitting solution. Please check back soon for the next installment! Peace and chow, Ranok

Open Server Platform Version 0.3 Released!

Today I finally got around to going through and testing the RC for version 0.3 of Open Server Platform. Everything seemed to work as planned, except for a few known issues that will be fixed in the next release, which should be coming down the pipe soon.

Improvements:

  • Added replication nodes to increase fail-safe reliability
  • Added commands to the administration console to stop and live migrate applications
  • Can use configuration files to assist in the start up of OSP
  • Many bug fixes and documentation updates
  • The HTTPd servlet example now supports large files and the HEAD command

So, please check it out and let me know what you think!

Peace and chow,

Ranok

OSP Version 0.2 Released!

After less than a week OSP has reached another milestone in it’s development! Version 0.2 was just released with a few big fixes, performance improvments and added functionality. Also the documentation has been improved and made consistant with the code.

The most important bug fix was a performance issue that caused a slowdown over time, this has been fixed and along with it, the performance and scalability has increased! The biggest feature added was support for UDP (and soon SCTP, and SSL) which can be set without changing the code! Imagine writing a server and deploying it using TCP, UDP, SSL and SCTP, just by changing one line in the XML servlet.

Setup should be simple too, just download from Google Code, extract, make and then compile your servlet. Now on the first node, run ./start-osp.sh and then you can telnet to port 9876 and administer your cluster.

Links:

Enjoy,

Ranok

OSP v0.1 Released!

I’m happy to announce that my OSP version has gotten stable enough to release version 0.1! While still a very temperamental system, it is showing true promise. It’s gotten to the point where on the first node, you edit a configuration file and run start-osp.sh and on each of the other nodes, just copy join.sh and edit it on each machine. After it’s run on each, you can telnet into the first machine and start applications which all have shared state across the cluster.

For more information, check out the home page, the Google Code page and the Google Group.

Open Server Platform Update

    For those of you who don’t know, I’ve been working on a project making a server environment that only requires the server logic to be coded, the rest (high availability, load balancing, etc…) is all taken care of by my project. However, before I can start on the server itself, I need to make a redundant file-system that lets distributed nodes keep synced up. It’s almost at a prototype stage where I can demo to the world. I put up a website and a Google code project with a SVN repository.

It’s written entirely in Erlang, and using the Mnesia database for the meta-data storage. It’s licensed under the MIT license so it’s business friendly and open-source.

Peace and chow,

Ranok