So, I decided that a good first step would be to implement a simple TCP server in Erlang, extending what I introduced in previous posts. There are various examples out there, but I decided to follow “Building a Non-blocking TCP server using OTP principles” on TrapExit. Not too hard to do, and theoretically implements an echo server. However, one gotcha is that when you setup the socket, you’ll use lots of {packet, 2} options and I didn’t realize at first that this is an application-layer thing that helps Erlang decide when to throw you an event. So you will not be able to just log into telnet and bounce things off of your server, because telnet won’t know to include the packet length. It’s easy enough to do this in Perl, by prepending a packed message length to your message, but be forewarned. The easiest thing to do is to replace it with {packet, line} and then it will maybe do what you’re expecting.

Frankly, I found this whole thing a bit annoying, because it wasn’t clear from the documentation that gen_fsm was going to magically receive messages from the socket when data arrived. But I eventually sorted this out and it seems to work okay.

Now what I want to do is to implement a simple XML-based protocol for my game. The idea is that this will fit well with the finite state machine approach, and allows me to extend it to include new functionality pretty easily. It’s not very good for bandwidth usage, but I don’t expect to send much data this way. ejabberd is a good example of XML streaming in Erlang, and there are at least two libraries out there for parsing: xmerl (User’s Guide) and erlsom. Since I want streamed parsing, I’ll need an event-based parser with something like SAX. Neither of them have much documentation about event-driven usage (lack of documentation is starting to be my biggest problem with Erlang), but this thread would imply that I’m better off with Erlsom. Trying it out, however, it wouldn’t work without a complete document. I want to implement something like the “stanzas” of XMPP.

Of course, for some reason, doing even the simplest things in Erlang is like pulling teeth. I thought I’d play with xmerl to see if might do what I want. I look at the sparse documentation, and at this blog entry and write the simplest possible program for parsing an xml file.

-module(lame).
-include_lib(“xmerl/include/xmerl.hrl”).
-export([ parsemo/0 ]).

parsemo() ->
{Xml, Rest} = xmerl_scan:file(“foo.xml”),
io:format(“~p ~p”, [Xml, Rest]).

foo.xml is just about the simplest thing you could imagine:

<?xml version=”1.0″ encoding=”utf-8″ ?>
<foo>
bar
</foo>

I try to run this thing, and get this:

Erlang (BEAM) emulator version 5.5.5

Eshell V5.5.5 (abort with ^G)
1> c(lame).
{ok,lame}
2> lame:parsemo().

=ERROR REPORT==== 18-Feb-2008::16:44:14 ===
Error in process <0.31.0> with exit value: {undef,[{xmerl_scan,file,["foo.xml"]},{lame,parsemo,0},
{erl_eval,do_apply,5},{shell,exprs,6},{shell,eval_loop,3}]}

** exited: {undef,[{xmerl_scan,file,["foo.xml"]},
{lame,parsemo,0},
{erl_eval,do_apply,5},
{shell,exprs,6},
{shell,eval_loop,3}]} **

What the hell? For one thing, there is almost no information about what is wrong. This is a huge failing on the part of Erlang. The error messages are ridiculous. I assume that the problem is that it doesn’t know about xmerl and can’t resolve the function call. It managed to load it just fine with the include_lib, but it doesn’t seem that it’s finding xmerl_scan:file. I don’t even know where to begin, to figure out how to get Erlang to find the module. I mean, am I insane? This is the sort of thing that makes a tool a no-go for me. I could write a streaming XML parser in Perl in about ten minutes. I know this because I’ve done it before. XML parsing is the sort of thing that should never have to be implemented in a modern development toolset – there should be clearly documented libraries with standard interfaces.

Taking a deep breath and looking at Section 6.7 of Joe Armstrong’s book, I see that this cryptic error message indicates one of four possible failure modes in trying to find xmerl_scan:file: 1) no module xmerl_scan, 2) xmerl_scan hasn’t been compiled, 3) xmerl_scan.beam can’t be found, 4) there are multiple versions of xmerl_scan floating around in the path. I was under the impression that xmerl was included in the standard libraries. I installed Erlang 5.5.5 via apt-get erlang-base-hipe (Ubuntu Gutsy), and it would seem that xmerl_scan is around, inasmuch as man xmerl_scan gives me the “docs.” However, I’m not able to actually use it.

After much wringing of hands, I installed every erlang-related package I could find via apt-cache and eventually it seemed to work. I have no idea what the problem was. Now my silly parser outputs:

{xmlElement,foo,
foo,
[],
{xmlNamespace,[],[]},
[],
1,
[],
[{xmlText,[{foo,1}],1,[],”\n bar\n”,text}],
[],
“.”,
undeclared} []

which is at least a step in the right direction. When I try to run it on a partial string, however, we fail. I think this is the behavior that xmerl intends to have. I’m starting to think that implementing whatever ejabberd does is going to be the only way to resolve this, unfortunately. So, I guess that’ll be next time.

There are three things that I’ve had to solve over and over (or at least cut and paste over and over), in every development project I’ve ever done. These things are: build management, configuration, and logging. These things have led me down every path one can imagine, in C/C++, Perl, Python and Java: make, ant, MakeMaker, Boost, log4j, log4cpp, Commons Configuration, whatever, I’ve probably tried to use it in a project at some point, and maybe rolled my own version as well.

So, the first thing I want to learn is how to manage a large project correctly in Erlang. There seem to be two concepts for this sort of thing: Applications and Releases. My understanding is that Applications let you bundle processes together into a coherent whole, while Releases let you make something “turn-key” and ready to run. I should also mention that I want this to be a distributed project. There are two basic reasons why you might want to have a distributed application: 1) reliability, and 2) scalability. For the hypothetical MMO framework we want, we need both. Sure, we don’t need the “Nine Nines” of the telecom world, but if the MMO is successful people will want to get their fix, so maybe we want three nines. Scalability is much more important. Since we’re fantasizing anyway, let’s imagine that we want to be able to have around a million users on an unsharded world. Obviously this is going to require a pretty hefty cluster of the sort of machines that are cheap to maintain and replace. Anyway, regardless of the stupidity of these numbers, the point is that NZ needs to be a distributed application.

In practical terms, the most useful things I’ve seen for getting started have been this page by Pete Kazmier, which talks about building an application using OTP principles, and this page at TrapExit, which has been down for several days. Also, there seems to be a lot of stuff going on at Erlware. The goal there seems to be to make a unified system like autotools for making Erlang code available to the public. That is a noble goal to be sure, but at this point Sinan and friends just seem like another learning curve to climb. In addition, the only tutorial around for the build system claims that the tool is broken.

The TrapExit page says

You will need to download the reference build system for this tutorial, this can be found at www.erlware.org under downloads otp_base-vsn

but it doesn’t look to me like there’s a downloads section at erlware.org. After some digging it appears that you can get the OTP base from Google Code. To complicate matters, there are two versions there. It seems that otp_base-R1-1.tgz is the stable release. I untarred this file and got an otp directory and various directories beneath it. I cd otp/tools/utilities and execute ./appgen north_zulch nz (the first argument is the application name and the second argument is going to be prefixed to everything), which spewed out a bunch of osbcenities:

[: 11: ==: unexpected operator
[: 24: ==: unexpected operator
[: 18: ==: unexpected operator
[: 32: ==: unexpected operator
-- SNIP --
[: 18: ==: unexpected operator
[: 32: ==: unexpected operator
rm: missing operand
Try `rm --help' for more information.

north_zulch has been generated and placed under lib/north_zulch
north_zulch_rel has been generated and placed under release/north_zulch_rel

I'm on Ubuntu Gutsy Gibbon, and it turns out that these are because the shell scripts start with #!/bin/sh when they really want #!/bin/bash. The error about rm: missing operand is because there's a command to remove Subversion directories and there aren't any in the tarball. So I went through and changed the shebang line in otp/tools/.appgen/subsitute.sh, otp/tools/.appgen/rename.sh, and otp/tools/utilities/appgen. Then I ran my command again and got

moving blank_app_rel.config.src to north_zulch_rel.config.src
moving blank_app_rel.rel.src to north_zulch_rel.rel.src
replacing %%APP_NAME%% with north_zulch in north_zulch_rel/north_zulch_rel.rel.src
moving ba_server.erl to nz_server.erl
moving ba_sup.erl to nz_sup.erl
moving blank_app.app.src to north_zulch.app.src
moving blank_app.appup.src to north_zulch.appup.src
moving blank_app.erl to north_zulch.erl
/home/rpa/Desktop/otp-diff2/tools/.appgen
replacing %%APP_NAME_UPPER_CASE%% with NORTH_ZULCH in north_zulch/Makefile
replacing %%APP_NAME_UPPER_CASE%% with NORTH_ZULCH in north_zulch/vsn.mk
replacing %%APP_NAME%% with north_zulch in north_zulch/src/Makefile
replacing %%APP_NAME_UPPER_CASE%% with NORTH_ZULCH in north_zulch/src/Makefile
replacing %%PFX%% with nz in north_zulch/src/Makefile
replacing %%APP_NAME%% with north_zulch in north_zulch/src/north_zulch.erl
replacing %%PFX%% with nz in north_zulch/src/north_zulch.erl
replacing %%PFX%% with nz in north_zulch/src/nz_sup.erl
replacing %%APP_NAME%% with north_zulch in north_zulch/src/nz_sup.erl
replacing %%PFX%% with nz in north_zulch/src/nz_server.erl
replacing %%APP_NAME%% with north_zulch in north_zulch/src/nz_server.erl
north_zulch has been generated and placed under lib/north_zulch
north_zulch_rel has been generated and placed under release/north_zulch_rel

which I am assuming means success. As promised, this created directories otp/lib/north_zulch and otp/release/north_zulch/rel with various things beneath them. A significant file here is otp/lib/north_zulch/vsn.mk which contains the version number. I'm going to leave it at 1.0 for now. In that same directory are include which are for shared macro and record definitions, and src where all the Erlang code files go. Significantly, I'm going to include north_zulch and nz in everything I write here, but you'll obviously want to change that for your application.

In the src directory are various stubs. The north_zulch.erl defines the "application callback module." You can think of a callback module as an approximation to deriving from an asbract base class with virtual functions. The function names are there, and they'll get called under certain well-defined circumstances. It's up to you to make them do something reasonable. There are three functions exported so that the application OTP module knows what to do with us. start/2 is called to get us going, shutdown/0 is called to get us to end, and stop/1 is called when we're all done. We look at these and compare them to the docs here.

start(Type, StartArgs) ->
    case nz_sup:start_link(StartArgs) of
	{ok, Pid} ->
	    {ok, Pid};
	Error ->
	    Error
    end.

We get two arguments. Type is an atom that will generally be normal for non-distributed applications. The distributed case is more complicated, so we'll look at that later down the road. We see that the default thing to do is to just try to run nz_sup:start_link(StartArgs) which starts up a Supervisor, regardless of the atom in Type. This appears to be the only value that the spec identifies in the non-distributed case, so we'll roll with it. In any case, if all goes well, we have to return {ok, Pid, State}. If State is left out, then it just uses []. We'll ignore State for now. Pid is the process id of the main Supervisor, the boss. If start_link (which we'll talk about in a sec) goes well, it gives us Pid. The stop/1 function is pretty trivial and is called after everything is done. As far as I can tell, shutdown/0 is not actually part of the Application interface, and is just a convenient way for you to shut down your application if you want to. It just goes up and tells the Application module that it wants to be turned off:

shutdown() ->
    application:stop(north_zulch).

With that big file under our belt, we now look at nz_sup.erl which contains the code for the supervisor. The Supervisor (also, the spec) is a great idea to build as a system primitive. I think I've implemented it at least five times in various projects. The basic idea is, manage some underlings and make sure they do their thing, while isolating yourself from their errors. If one fails, start up another, modulo your configurable policy about such things. The docs have more info on restart behaviors. We export, for our own purposes, the function start_link/1, which got called by start/1 in the previous file:

start_link(StartArgs) ->
    supervisor:start_link({local, ?SERVER}, ?MODULE, []).

This just turns around and tells the Supervisor API that we want to be fired up as a supervisor callback module. supervisor:start_link/3 has three arguments. The first is a tuple that tells the Supervisor module who we are. The first bit of the tuple can be local or global. The second is the name of the appropriate callback module, which in this case is ?SERVER. What does that mean? Well, things that start with a question mark in Erlang are macros. Up at the top of the file is

-define(SERVER, ?MODULE).

which defines it. ?MODULE is a special one with the module name. Back to start_link/3, we also specify the name of the callback module - this one - using ?MODULE, and any arguments we want to pass along to init/1, which is the other significant function we need to export to use the supervisor:

init([]) ->
    RestartStrategy    = one_for_one,
    MaxRestarts        = 1000,
    MaxTimeBetRestarts = 3600,

    SupFlags = {RestartStrategy, MaxRestarts, MaxTimeBetRestarts},

    ChildSpecs =
	[
	 {nz_server,
	  {nz_server, start_link, []},
	  permanent,
	  1000,
	  worker,
	  [nz_server]}
	 ],
    {ok,{SupFlags, ChildSpecs}}.

This is pretty much where all the guts of the supervisor's behavior are. The one argument you get is whatever you handed to start_link/3 before, so you could pass around various application data if you wanted. We're not worrying about this at the moment, though, and we're just going to look at what this gets up to. First off, we define some variables that control the supervisor's behavior. RestartStrategy determines what restarting should occur should a child fail. If MaxRestarts failures occur before MaxTimeBetRestarts seconds, then all children are cancelled and the supervisor terminates itself. We construct a list of child specifications, that in this case only contains one element: the specs for managing the nz_server process. Within this specification are:

	 {nz_server,
	  {nz_server, start_link, []},
	  permanent,
	  1000,
	  worker,
	  [nz_server]}

The first component is a name that the supervisor uses to keep track of this child. In this case, we use the term nz_server, since there's only one. The second component is a Module-Function-Arguments triplet that tells the supervisor what function should be called in the child. We're going to look at this later, but the function is start_link/0 in the nz_server module. The next argument can be permanent, transient, or temporary and specifies what should happen if it dies. The fourth argument specifies how long we should wait around for a termination signal to be handled by a child. We can create trees of supervisors, so the fifth argument specifies whether this is a worker or a supervisor itself. The final element will generally be a single-element list with the name of the callback module of the worker, in this case nz_server. We return a tuple with these two specs together and they tell the Supervisor how it should act, and how is should initialize its children.

The final source file we examine is nz_server.erl, which implements the application-specific work. At the very top of the file, we see

-behaviour(gen_server).

which means that it implements the gen_server (also, here) behaviour as a callback. The callbacks we need to define for this module to do its thing are init/1, handle_call/3, handle_cast/2, handle_info/2, terminate, and code_change/3. Again, if you come from a background like C++ or Java, you can think of these approximately as virtual event-handling functions. Rather than an onMouseClick() type override, we have various functions that are called when the server receives Erlang messages. These are all exported so that the gen_server module can get to them.

There are also two interface functions that are defined that don't really have anything to do with gen_server. The first is start_link/0, which we mentioned before. This is called by the Supervisor to get things going:

start_link() ->
    gen_server:start_link({local, ?SERVER}, ?MODULE, [], []).

You can see that we immediately just hand control off to the generic server. It takes a couple of arguments. The first is a tuple that is like the previous start_link in that we're telling it to look for this module and to register a name locally. The other "local" function is stop/0 which provides an interface to shutting down gen_server by sending it an asynchronous message.

To actually do anything interesting, we'll need to implement the gen_server functions. I'll leave that to the next installment. We'll also look at releases, configurations and logging, which is what we said we were after in the first place!

For scalability, and for an excuse to learn something new, I thought I’d play with Erlang. Erlang is a functional programming language that is completely geared toward concurrency. It’s not so fast with the number-crunching we’d expect to do with a game for physics and collision detection, but it is really well-suited for having zillions of semi-independent entities wandering around. Hopefully we can figure out a way to offload hard core number crunching to C via an FFI interface of some kind, but we’ll cross that bridge when we get to it. For now, I’m just trying to learn how to build a real-life application in Erlang.

Most of the documentation on Erlang seems to revolve around learning functional programming. I feel pretty good about FP already, due to having taken 6.001 back in the day. Tutorials about implementing the factorial function in Erlang aren’t really what I’m looking for. I’ve ordered Programming Erlang from Amazon and it’s due to arrive on Friday, so hopefully that’ll get me going. Stay tuned and maybe this will all be useful to you, too.