Tag Archives: erlang

Introduction to Mnesia I


Warning: Invalid argument supplied for foreach() in /home/public/wp-content/plugins/ionhighlight/Text/Highlighter/Renderer/Html.php on line 308

Today in my Advanced Concepts in Operating Systems class I led the discussion on the Mnesia paper from PADL’99, while this paper has numerous typos it does do an excellent job highlighting the features and advantages of Mnesia. For those of you who are unaware, Mnesia is a distributed, fault-tolerant object DBMS written in Erlang. One thing about Mnesia that I have found to be lacking is a tutorial written for the lay person from the ground up, this gap I intend to try and fill. This multi-segment tutorial assumes you have knowledge of Erlang, and the basic ┬áconcepts of manipulating data with DBMSes, other than that, I hope to provide enough information and code to demystify a fairly complex system. However, I still am on the road to mastery, so if I make any errors, or you have any tips for improvement, I’d be happy to add them in.

To get started, start up the Erlang shell (erl) with a name, I will use -sname foo in the following examples. Below is a transcript of starting up and creating a disk-based

ranok@orion:~/Desktop$ erl -sname foo
Erlang R13B01 (erts-5.7.2) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]
 
Eshell V5.7.2  (abort with ^G)
(foo@orion)1> mnesia:create_schema([node()]).
ok
(foo@orion)2> mnesia:start().
ok

The most important call made here, create_schema, takes a list of nodes to replicate the schema table to on disk. You can add additional disk-based nodes or ram-based copies later (we’ll get to the details later). After you’ve created the schema (this will make a folder for all the Mnesia table data), you can start the application with the start function.

Now that we have the database running, we need at least one table to store the data in, we will start with a very simple record to just store simple key/value data. The nice thing about Mnesia, is that the data we store can be pretty much anything, from a simple atom to a function. We will start learning the basics from the mnesia_test module, which I have uploaded here.

The first few lines of the module start off like any Erlang code, a module declaration, what functions to export, including the QLC (Query List Comprehensions) include file (you may need to find it for your system) and a record definition:

-record(data, {key, value}).

Which will define our record for the table also named data. In the function setup_and_start/0, we tie in what we already went over with the create_table function, which in our case looks like

mnesia:create_table(data, [{disc_copies, [node()]}, {attributes, record_info(fields, data)}])

The create_table function has a number of options, the most basic of which we will deal with at the moment: the name of the new table, where and how the table will be stored and what fields the table has (this code used the record_info() function to pull those out of the data record for us). Now that we have our table, we need a way to get the data in and out of it.

Many databases provide the ability for multiple queries to be joined into one transaction to be executed atomically. Mnesia is no different, but for the most part, all queries are executed through the transaction manager (there is a dirty interface which will be discussed later), this makes working in a distributed environment much easier. The way to perform a transaction in Mnesia is to pass a fun to the mnesia:transaction() function that will atomically run.

The actual function to enter data (both insert and update) is write(Record). We wrap this into the mnesia_test:insert(Key, Val) function displayed below:

insert(Key, Val) ->
Record = #data{key = Key, value = Val},
F = fun() ->
mnesia:write(Record)
end,
mnesia:transaction(F).

To now retrieve this data back from the database, the read function is now used, the read function takes two arguments: the table name (in this case, data) and the key to retrieve. The mnesia_test retrieve function wraps this nicely for us, and is shown below:

retrieve(Key) ->
F = fun() ->
mnesia:read({data, Key})
end,
{atomic, Data} = mnesia:transaction(F),
Data.

mnesia:transaction will either return {aborted, Reason} or {atomic, Rows}, where Rows is a list of all the retrieved data. If the key we tried to retrieve could not be found, then it will return an empty list.

Say however, we want to search the table for certain values that are not the index of the table. For that there is the matching functions, the simplest of them is the match_object whose usage can be seen here:

search(Val) ->
F = fun() ->
mnesia:match_object(#data{key = '_', value = Val})
end,
{atomic, Data} = mnesia:transaction(F),
Data.

As you can see, simply fill in all the values that are known and that you want to search for, and use the ‘_’ unmatched value for all the other values. This transaction will return the same forms as the read transaction.

There is another method for filtering through Mnesia tables, which is very similar to the list comprehensions builtin to Erlang which is called QLC. The QLC version of the above function is below:


search_qlc(Val) ->
F = fun() ->
qlc:eval(
qlc:q(
[X || X <- mnesia:table(data), X#data.value == Val] )) end, {atomic, Data} = mnesia:transaction(F), Data.[/code] What the query here is doing is is returning a list of Xs where every possible X comes from our data table, and X#data.value == Val. This should be very intuitive for those of you who are familiar with list comprehensions. What qlc:q() does is form a query handle (much like a function object) which gets evaluated by qlc:eval() inside of our transaction object. Again, this will return the same values from mnesia:transaction. Well, that about covers the basics of Mnesia, you now should be able to setup Mnesia on your computer, create a table and insert/retrieve data from it. In the next installment, we will look at distributed Mnesia and the dirty interface, which provides faster queries by bypassing the transaction manager. After that we will put all of what we've learned into creating a system that will take advantage of Mnesia and give a pseudo real-world problem a fitting solution. Please check back soon for the next installment! Peace and chow, Ranok

Using Inets – Erlang’s Builtin Web Daemon

A feature I’ve been meaning to add to Open Server Platform for a while is a web management system, where an administrator can login and manage the cluster and the servlets running on it. I’d like there to be a user friendly interface for administrators to start, stop and migrate servlets across the different nodes of the system and a way to upload a servlet file and have it compiled and distributed across the cluster.

The simplest way to start serving web content with Erlang is to use the inets server and the httpd service. This is a HTTP/1.1 server built into the Erlang distribution that supports some more advanced features, most interesting of all, the ability to use Erlang to dynamically generate content. It is however very poorly documented, and there are a few very annoying things I came across that I’m posting to hopefully help anyone else trying to get it working.

  1. The order of modules in the {modules, []} directive matters, if you want to have mod_dir work, it needs to be specified *AFTER* the mod_alias.
  2. The logging is rather horrid, the transfer.log will not log anything except for HTTP 200 for every request, even if it failed.
  3. You must specify {bind_address, any} in the configuration to use the httpd:reload_config function, otherwise it will return {error, not_started}
  4. If you just want to server static content, you will need at a minimum the following modules: mod_get, mod_head, mod_log, mod_actions and mod_range. However, adding mod_alias is recommended along with the {directory_index, ["index.html"]} directive to stop it from failing (HTTP 500) on a directory request.
  5. To use dynamic content, create a module that exports callbacks of the form: function(SessionID, _Env, _Input). To write Str back to the client, use the mod_esi:deliver(SessionID, Str) function.

I hope that this helps out!

Peace and chow,

Ranok

Open Server Platform Version 0.3 Released!

Today I finally got around to going through and testing the RC for version 0.3 of Open Server Platform. Everything seemed to work as planned, except for a few known issues that will be fixed in the next release, which should be coming down the pipe soon.

Improvements:

  • Added replication nodes to increase fail-safe reliability
  • Added commands to the administration console to stop and live migrate applications
  • Can use configuration files to assist in the start up of OSP
  • Many bug fixes and documentation updates
  • The HTTPd servlet example now supports large files and the HEAD command

So, please check it out and let me know what you think!

Peace and chow,

Ranok

Back in the Groove

After taking off the afternoon due to a migraine, and napping for a few hours until it passed, I was looking over some old projects, and decided to get motivated to work on my various Erlang hackery projects. I checked out a fresh copy of Open Server Platform on my new computer, and decided to hack on it some more. There are a number of loose ends I’d like to tie up before the 0.3 release, but, before I could get down to coding, SVN get my so frustrated that I could no longer deal with it and switched to git, creating a new repository on Github. I will keep the Google Code repository the ‘defacto’ repository, where the safer code gets committed, but use Github for the more bleeding edge development due to it’s simplified branching and merging (among other things).

In the new git setup, I have two branches (aside from master, which follows the SVN repository): otp and no-otp. The no-otp version is currently the stable code that runs just fine, but doesn’t take advantage of Erlang’s OTP framework. The otp branch is the more cutting edge OTP aware version, which I hope to fully migrate to soon. With full support for OTP, I should be able to use an already existing distribution platform, and more battle tested redundancy.

This evening, I fixed a long standing bug in the example HTTPd where the server would crash when opening large files due to a shortcut I took with first implementing the server. Originally, the server would read the entire file into a string and then send that to the client, as you can probably see, there is a problem when the server tries to read in a 3.9 GB file (my test file). Now, my servlet takes a much more sane approach, read in the file 1 kilobyte at time, sending that to the client before reading more. This new approach works perfectly, though I had to add some messiness for handling CGI/PHP files and different MIME types. I also added support for the HTTP HEAD command.

Peace and chow,

Ranok

Next Semester Projects

Seeing as I’m going to have no homework for the next semester (!!) I’m hoping to get some more projects underway in my down time. I’m going to use this post as a dump of my current interests and as a road-map for where I want to go in the next few months. I’m sure this will change as time goes on, but I need to start somewhere!

Projects in progress:

  • FANG – I’d like to polish the multi-processing system, add namespaces, and tie it into the Erlang stdlib. Also add macros (real macros!) to FANG.
  • OSP – I’m going to be using OSP as a basis for LadieBug (mentioned below). This will hopefully give me perspective into what I should work on in OSP to make it more friendly for development.

New projects:

  • I’d like to take advantage of LaunchPad’s ‘Personal Package Archive’ system to host a few of my own Ubuntu packages, namely a more up-to-date version of Erlang’s OTP system.
  • LadieBug – To take advantage of OSP’s distributed data store, I’d like to implement my own caching, recursive DNS server that would be both DNSSEC and DNSCurve compliant and would share the cache over the entire cluster, making it more scalable and reduce the number of needless queries.

That’s about all I can think of at the moment, if anyone would like to help me on any of these, feel free to comment below and we can get in touch.

Peace and chow,

Ranok