To get started, start up the Erlang shell (erl) with a name, I will use -sname foo in the following examples. Below is a transcript of starting up and creating a disk-based

ranok@orion:~/Desktop$ erl -sname foo
Erlang R13B01 (erts-5.7.2) [source] [smp:2:2] [rq:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.2  (abort with ^G)
(foo@orion)1> mnesia:create_schema([node()]).
ok
(foo@orion)2> mnesia:start().
ok

The most important call made here, create_schema, takes a list of nodes to replicate the schema table to on disk. You can add additional disk-based nodes or ram-based copies later (we’ll get to the details later). After you’ve created the schema (this will make a folder for all the Mnesia table data), you can start the application with the start function.

Now that we have the database running, we need at least one table to store the data in, we will start with a very simple record to just store simple key/value data. The nice thing about Mnesia, is that the data we store can be pretty much anything, from a simple atom to a function. We will start learning the basics from the mnesia_test module, which I have uploaded here.

The first few lines of the module start off like any Erlang code, a module declaration, what functions to export, including the QLC (Query List Comprehensions) include file (you may need to find it for your system) and a record definition:

-record(data, {key, value}).

Which will define our record for the table also named data. In the function setup_and_start/0, we tie in what we already went over with the create_table function, which in our case looks like

mnesia:create_table(data, [{disc_copies, [node()]}, {attributes, record_info(fields, data)}])

The create_table function has a number of options, the most basic of which we will deal with at the moment: the name of the new table, where and how the table will be stored and what fields the table has (this code used the record_info() function to pull those out of the data record for us). Now that we have our table, we need a way to get the data in and out of it.

Many databases provide the ability for multiple queries to be joined into one transaction to be executed atomically. Mnesia is no different, but for the most part, all queries are executed through the transaction manager (there is a dirty interface which will be discussed later), this makes working in a distributed environment much easier. The way to perform a transaction in Mnesia is to pass a fun to the mnesia:transaction() function that will atomically run.

The actual function to enter data (both insert and update) is write(Record). We wrap this into the mnesia_test:insert(Key, Val) function displayed below:

insert(Key, Val) ->
Record = #data{key = Key, value = Val},
F = fun() ->
mnesia:write(Record)
end,
mnesia:transaction(F).

To now retrieve this data back from the database, the read function is now used, the read function takes two arguments: the table name (in this case, data) and the key to retrieve. The mnesia_test retrieve function wraps this nicely for us, and is shown below:

retrieve(Key) ->
F = fun() ->
mnesia:read({data, Key})
end,
{atomic, Data} = mnesia:transaction(F),
Data.

mnesia:transaction will either return {aborted, Reason} or {atomic, Rows}, where Rows is a list of all the retrieved data. If the key we tried to retrieve could not be found, then it will return an empty list.

Say however, we want to search the table for certain values that are not the index of the table. For that there is the matching functions, the simplest of them is the match_object whose usage can be seen here:

search(Val) ->
F = fun() ->
mnesia:match_object(#data{key = '_', value = Val})
end,
{atomic, Data} = mnesia:transaction(F),
Data.

As you can see, simply fill in all the values that are known and that you want to search for, and use the ‘_’ unmatched value for all the other values. This transaction will return the same forms as the read transaction.

There is another method for filtering through Mnesia tables, which is very similar to the list comprehensions builtin to Erlang which is called QLC. The QLC version of the above function is below:

search_qlc(Val) ->
F = fun() ->
qlc:eval(
qlc:q(
[X || X <- mnesia:table(data), X#data.value == Val]
))
end,
{atomic, Data} = mnesia:transaction(F),
Data.

What the query here is doing is is returning a list of Xs where every possible X comes from our data table, and X#data.value == Val. This should be very intuitive for those of you who are familiar with list comprehensions. What qlc:q() does is form a query handle (much like a function object) which gets evaluated by qlc:eval() inside of our transaction object. Again, this will return the same values from mnesia:transaction.

Well, that about covers the basics of Mnesia, you now should be able to setup Mnesia on your computer, create a table and insert/retrieve data from it. In the next installment, we will look at distributed Mnesia and the dirty interface, which provides faster queries by bypassing the transaction manager. After that we will put all of what we’ve learned into creating a system that will take advantage of Mnesia and give a pseudo real-world problem a fitting solution. Please check back soon for the next installment!

Peace and chow,

Ranok

The simplest way to start serving web content with Erlang is to use the inets server and the httpd service. This is a HTTP/1.1 server built into the Erlang distribution that supports some more advanced features, most interesting of all, the ability to use Erlang to dynamically generate content. It is however very poorly documented, and there are a few very annoying things I came across that I’m posting to hopefully help anyone else trying to get it working.

1. The order of modules in the {modules, []} directive matters, if you want to have mod_dir work, it needs to be specified *AFTER* the mod_alias.
2. The logging is rather horrid, the transfer.log will not log anything except for HTTP 200 for every request, even if it failed.
3. You must specify {bind_address, any} in the configuration to use the httpd:reload_config function, otherwise it will return {error, not_started}
4. If you just want to server static content, you will need at a minimum the following modules: mod_get, mod_head, mod_log, mod_actions and mod_range. However, adding mod_alias is recommended along with the {directory_index, ["index.html"]} directive to stop it from failing (HTTP 500) on a directory request.
5. To use dynamic content, create a module that exports callbacks of the form: function(SessionID, _Env, _Input). To write Str back to the client, use the mod_esi:deliver(SessionID, Str) function.

I hope that this helps out!

Peace and chow,

Ranok

Improvements:

• Added replication nodes to increase fail-safe reliability
• Added commands to the administration console to stop and live migrate applications
• Can use configuration files to assist in the start up of OSP
• Many bug fixes and documentation updates
• The HTTPd servlet example now supports large files and the HEAD command

So, please check it out and let me know what you think!

Peace and chow,

Ranok

In the new git setup, I have two branches (aside from master, which follows the SVN repository): otp and no-otp. The no-otp version is currently the stable code that runs just fine, but doesn’t take advantage of Erlang’s OTP framework. The otp branch is the more cutting edge OTP aware version, which I hope to fully migrate to soon. With full support for OTP, I should be able to use an already existing distribution platform, and more battle tested redundancy.

This evening, I fixed a long standing bug in the example HTTPd where the server would crash when opening large files due to a shortcut I took with first implementing the server. Originally, the server would read the entire file into a string and then send that to the client, as you can probably see, there is a problem when the server tries to read in a 3.9 GB file (my test file). Now, my servlet takes a much more sane approach, read in the file 1 kilobyte at time, sending that to the client before reading more. This new approach works perfectly, though I had to add some messiness for handling CGI/PHP files and different MIME types. I also added support for the HTTP HEAD command.

Peace and chow,

Ranok

Projects in progress:

• FANG – I’d like to polish the multi-processing system, add namespaces, and tie it into the Erlang stdlib. Also add macros (real macros!) to FANG.
• OSP – I’m going to be using OSP as a basis for LadieBug (mentioned below). This will hopefully give me perspective into what I should work on in OSP to make it more friendly for development.

New projects:

• I’d like to take advantage of LaunchPad’s ‘Personal Package Archive’ system to host a few of my own Ubuntu packages, namely a more up-to-date version of Erlang’s OTP system.
• LadieBug – To take advantage of OSP’s distributed data store, I’d like to implement my own caching, recursive DNS server that would be both DNSSEC and DNSCurve compliant and would share the cache over the entire cluster, making it more scalable and reduce the number of needless queries.

That’s about all I can think of at the moment, if anyone would like to help me on any of these, feel free to comment below and we can get in touch.

Peace and chow,

Ranok

Over break, I’m hoping to add enough other features to warrant a 0.2 release, which I’d like to have support for soft-processes (ala Erlang), multi-node support and transactional shared memory (using Mnesia). I would also like to add permissions and process jailing for the soft-processes to allow for running untrusted code in a sandbox of sorts.

I’ve started a very simple (and ugly) site to put my progress and releases. You can check it out here and please comment with suggestions or other features you’d think would be a valuable addition. Good luck on everyone’s finals and projects! Have a wonderful break!

Peace and chow,

Ranok

Features I want FANG to have:

• Clean scheme/lisp syntax and macros
• Lightweight processes and multi-node support
• Arbitrary precision number support
• Process jailing and permission setting
• Simple networking and file I/O
• Others I’m sure I forgot…

Code Examples:

• (defvar ‘fib (lambda ‘(n) ‘(if (< n 3) (1) (+ (fib (- n 1)) (fib (- n 2)))))) – A plain recursive function that automatically will get memoized, so it’s rather speedy (use dirty-lambda to turn off the memoization)
• (defvar ‘max (lambda ‘(a b) ‘(if (> a b) (a) (b)))) – A simple max function used in findset
• (defvar ‘findset (lambda ‘(l) ‘(if (== (length l) 0) 0 (if (== 1 (length l)) (hd l) (max (+ (hd l) (if (== 2 (length l))) 0 (findset (tl (tl l)))) (findset (tl l))))))) – A function for a homework assignment to find the highest weighted independent set in a path
• (defvar ‘fact (lambda ‘(n) ‘(if (== 0 n) (1) (* n (fact (- n 1)))))) – A memoized factorial function

Peace and chow,

Ranok

Peace and chow,

Ranok

• r88 – Now will autosense is a node has a disk, and will determine whether or not to keep a copy of the applications table on the disk or just in RAM
• r89 – Made gen-join.pl which asks the user a bunch of questions to make a join script for easier setup
• r93 – When you first install OSP, you should run ./setup-osp.sh to initialize the database on the first/master node
• r96 – Added the add-backup-server, which will make a disk copy of all the tables to that backup server incase of node failure, there’s a backup copy

While thse changes are small, they are still very needed in creating an actual application. Currently, OSP is at r97, only 3 away from the big 100. I thought that we should have a mini OSP party when we reach r100, but since OSP is distributed, so should the party. So when you get word of the 100th commit, please raise a glass or eat a slice of cake.

The most important bug fix was a performance issue that caused a slowdown over time, this has been fixed and along with it, the performance and scalability has increased! The biggest feature added was support for UDP (and soon SCTP, and SSL) which can be set without changing the code! Imagine writing a server and deploying it using TCP, UDP, SSL and SCTP, just by changing one line in the XML servlet.

Setup should be simple too, just download from Google Code, extract, make and then compile your servlet. Now on the first node, run ./start-osp.sh and then you can telnet to port 9876 and administer your cluster.

Links:

• Open Server Platform home
• OSP Group
• OSP Code

Enjoy,

Ranok