Client process

First a little catching up.
Time and date parsing, I decided to use the definition from Erlangs calendar module which are:

date() = {Year, Month, Day}
  Year = int()
  Month = 1..12
  Day = 1..31

time() = {Hour, Minute, Second}
  Hour = 0..23
  Minute = Second = 0..59

Only I need millisecond precision for the time so i decided on this compatible format
{{Hour, Minute, Second}, Millis}

Code for generating these is easy using iolib:fread/2

    {ok,[Day,Month,Year],[]} = io_lib:fread("~2d~2d~2d",Date),
    {ok,[Hour,Minute,Second,Millis],[]} =io_lib:fread("~2d~2d~2d.~3d",Time),

I get a list of the parsed values which can be passed directly into the relevant tuples.

     {fixtime,{{Hour,Minute,Second},Millis}},
     {date,{Year+2000,Month,Day}}

Since last I noticed that the GPS device prior to getting a satellite fix returned empty lists in places where the code expected integers, floats or atoms, leading to errors in the call to list_to_integer for example, so I had to replace these with functions like these:

 safe_to_float([]) ->
    -1.0;
 safe_to_float(F) -> 
    list_to_float(F).

Now to the client process. I simple spawn a new process which will receieve messages from the driver and react only to gga messages as they are carrying the position information, then they format these in a google maps friendly way. The entire code goes like this:

start() ->
    spawn(fun() ->
                  gpsdriver:start(self()),
                  loop()
          end).
loop() ->
    receive
        {gga,
         {fixtime,_},
         {lattitude,[Lattitude,NS]},
         {longitude,[Longitude,EW]},
         {quality,_},
         {numberofsatellites,_},
         {horizdi,_},
         {altitude,_},
         {geoidheight,_},
         {timesincedgps,_},
         {dgpsstation,_}} ->
            io:format("~p~s,~p~s~n",[Lattitude,NS,Longitude,EW]);
        _ ->  true
    end,
    loop().

Note that in the start/0 function I pass the Pid (process id) of the newly created process to the gpsdriver, so that was changed to accept this and instead of outputting directly to standard output sends the tuples to this process using Erlang message sending mechanism Pid ! Record

The final code outputs messages like:

29.9792N,31.1343E
29.9792N,31.1343E
29.9792N,31.1343E


So this completes the gps project for me, only that I will make some performance measures to get an idea of the best way to parse the incoming messages.

However this has to be postponed to the new year, as a major holiday is approaching fast.

Best wishes for 2008 to everyone who bothers reading this.

Parsing take #3

I want to make more structured messages so I parse the lists using a function I call compose/1 for this I again use Erlangs superior pattern matching.

So for GPGSA the compose looks like:

compose(["GPGSA",A,B,C1,C2,C3,C4,C5,C6,C7,C8,C9,C10,C11,C12,D,E,F]) ->
    {gsa,
     {auto,list_to_atom(A)},
     {fix,list_to_integer(B)},
     {satellites,satellites([C1,C2,C3,C4,C5,C6,C7,C8,C9,C10,C11,C12],[])},
     {pdop,list_to_float(D)},
     {horizdi,list_to_float(E)},
     {vertdi,list_to_float(F)}};

This will only match the lists with "GPGSA" sentences (my GPS device only submits the sentences GPGSA,GPGGA,GPRMC,GPGSV).

The structured data I use are simple tuples, where I use Erlang atoms as identifiers for the fields, as you can see I also translate the integer data into integers (and for other messages to float values). A helper function is used so I do not repeat my self to much:

satellites([[] | _], Satellites) ->
   Satellites;
satellites([A | Rest], Satellites) ->
   satellites(Rest, [list_to_integer(A) | Satellites]).

The rest of compose for parsing the remaining sentences looks like:

compose(["GPGGA",A,B1,B2,C1,C2,D,E,F,G1,G2,H1,H2,I,J]) ->
   {gga,
    {fixtime,A},
    {lattitude,[list_to_float(B1)/100,list_to_atom(B2)]},
    {longitude,[list_to_float(C1)/100,list_to_atom(C2)]},
    {quality,list_to_integer(D)},
    {numberofsatellites,list_to_integer(E)},
    {horizdi,list_to_float(F)},
    {altitude,[list_to_float(G1),list_to_atom(G2)]},
    {geoidheight,[list_to_float(H1),list_to_atom(H2)]},
    {timesincedgps,list_to_float(I)},
    {dgpsstation,J}};

compose(["GPRMC",A,B,C1,C2,D1,D2,E,F,G,H1,H2]) ->
   {rmc,
    {fixtime,A},
    {status,list_to_atom(B)},
    {lattitude,[list_to_float(C1)/100,list_to_atom(C2)]},
    {longitude,[list_to_float(D1)/100,list_to_atom(D2)]},
    {speed,list_to_float(E)},
    {trackangle,list_to_float(F)},
    {date,G},
    {magneticvariation,[H1,H2]}};

compose(["GPGSV",A,B,C | SateliteData ]) ->
   {gsv,
    {numberofsentences,list_to_integer(A)},
    {number,list_to_integer(B)},
    {numberofsatellites,list_to_integer(C)},
    satellitedata(SateliteData, [])}.

I divide the longitude/lattitudes by 100 as I wish that they closely match the expected format for Google maps.

The GPGSV sentences are variable length so I use a helper function for parsing those:

satellitedata([D,E,F,G | SateliteData ], Satellites) ->
   satellitedata(SateliteData, 
                 [{sattelite,list_to_integer(D)},
                  {elevation,list_to_integer(E)},
                  {azimuth,list_to_integer(F)},
                  {snr,list_to_integer(G)} | Satellites]);
satellitedata([], Satellites) ->
   Satellites.

My final loop/1 looks like this:

loop(Port) ->
   receive
       {Port, {data, Data}} ->
           case Data of
               <<"$", Rest/binary>> ->
                   Tokens = tokenize (Rest),
                   Record = compose(Tokens),
                   io:format("~p~n", [Record]);
               _ ->
                   sl:setopt(Port, mode, line),
                   sl:update(Port)
           end,
           loop(Port);
       stop  ->
           exit(normal);
       {'EXIT', Port, Reason} ->
           exit({port_terminated, Reason})
   end.

The resulting messages are reported in the shell like:

{rmc,{fixtime,"214612.620"},
    {status,'A'},
    {lattitude,[29.9792,'N']},
    {longitude,[31.1343,'E']},
    {speed,9.00000e-2},
    {trackangle,81.9100},
    {date,"291107"},
    {magneticvariation,[[],[]]}}
{gga,{fixtime,"214613.620"},
    {lattitude,[29.9792,'N']},
    {longitude,[31.1343,'E']},
    {quality,1},
    {numberofsatellites,4},
    {horizdi,4.00000},
    {altitude,[146.2000,'M']},
    {geoidheight,[0.0000,'M']},
    {timesincedgps,0.00000e+0},
    {dgpsstation,"0000"}}
{gsa,{auto,'A'},
    {fix,3},
    {satellites,[2,23,20,31]},
    {pdop,7.30000},
    {horizdi,4.00000},
    {vertdi,6.00000}}
{rmc,{fixtime,"214613.620"},
    {status,'A'},
    {lattitude,[56.2778,'N']},
    {longitude,[10.2091,'E']},
    {speed,0.100000},
    {trackangle,100.610},
    {date,"291107"},
    {magneticvariation,[[],[]]}}

Note I still need to parse the dates and times into some nice representations.
I will next time make an additional process which will receive these messages and report only the position data in a format compatible with Google maps

Parsing take #2

NMEA messages are quite simple. They are simply comma separated values (some values might be absent) with start/end markers and a checksum.
The checksum is the xor of the bytes between the start and end markers, and is sent as a hex encoded value after the end marker. Like this:


$GPGSA,A,3,11,20,01,17,,,,,,,,,7.0,2.4,6.6*36\r\n


$ being the start marker * being the end marker and 36 hex being the checksum.

Last time I made the decision to match on the start of the message like:

           case Data of
               <<"$GPGSA,", Rest/binary>> ->
                   getparams(Rest);
               _ ->
                   io:format("~p~n", [Data])
           end,

This was actually not a good decission, because the NMEA messages are so alike I find it better to simply match on the start marker and receive a list of parsed values (tokens).

So my new case looks like this:

            case Data of
                <<"$", Rest/binary>> ->
                    Tokens = tokenize (Rest),
                    io:format("~p~n", [Tokens]);
                _ ->
                    sl:setopt(Port, mode, line),
                    sl:update(Port)
            end,

Now the second _ clause should never match, but when you call the stop/0 and then start/0 funktions the sl module sometimes (read bug) looses the information that line mode was requested, in these cases this clause will match and I force the line mode on the Port, which works around this.

Now to split the received binary into tokens I call tokenize/1, now I was hoping to keep the data in binaries and build a list of these, but constructing binaries by continuously appending bytes to them is not supported in erlang, so for a first go I will turn the binary into a list of strings (a possible solution to my first wish is to scan the binary, record the comma positions and use the split_binary built in function, I will try this later and compare the performance implications).

Now the tokenize will have two missions: tokenize the binary and verify the checksum

First tokenize/1:

tokenize(<<$G:8, Rest/binary>>) ->
    tokenize(Rest, [$G], [], $G).

Here I use that all NMEA messages starts with "G", so I can initialize the current token (second parameter) with a $G ($ is Erlangs syntax for characters) and the calculated checksum to the same value (fourth parameter) and the third parameter is the current list of parsed tokens which start out empty.

Now tokenize/4:

tokenize(<<$,:8 , Rest/binary>>, Token, Tokens, Sum) ->
    tokenize (Rest, [], [lists:reverse(Token) | Tokens], Sum bxor $,);
tokenize(<<$*:8 , Rest/binary>>, Token, Tokens, Sum) ->
    checksum (Rest, [lists:reverse(Token) | Tokens], [], Sum);
tokenize(<>, Token, Tokens, Sum) ->
    tokenize(Rest, [N | Token], Tokens, Sum bxor N).

The algorithm is simple when a comma is first in the binary, a token is finished so it is added to the list of parsed tokens (as I add characters at the beginning of the list in usual Lisp style, the lists:reverse needs to be called) and $, is added to the checksum. When a * is first we move on to parse the checksum, in all other cases the first byte in the binary is added to the current token and the checksum.

Now the checksum needs to be parsed and verified, the code below does this, note that the first clause with the empty binary should in theory never match, but again in some circumstances (read stop/start again) this can happen and this takes care of that. Otherwise it is pretty straight forward, the read checksum is converted to its numerical value using {_,ReadSum,_} = io_lib:fread("~16u",Chk),:

checksum(<<>>, _, _, _) ->
    {error};
checksum(<<$\r:8, Rest/binary>>, Tokens, Chk, Sum) ->
    checksum(Rest, Tokens, lists:reverse(Chk), Sum);
checksum(<<$\n:8>>, Tokens, Chk, Sum) ->
    {_,ReadSum,_} = io_lib:fread("~16u",Chk),
    case lists:nth(1,ReadSum) of
        Sum -> lists:reverse(Tokens);
        _ -> {error}
    end;
checksum(<>, Tokens, Chk, Sum) ->
    checksum(Rest, Tokens, [N | Chk ], Sum).

Thats it, we get the binary split into a list of tokens, next I will turn these lists into more structured records.

Parsing take #1

Erlang is not famous for its string representation, strings are just lists of characters and as such take up considerable memory, on 32 bit machines typically 64bit per character 32 for the characters value and 32 for the pointer to the next character (on 64bit machines that is doubled to 128 bit per character), ouch (for more details of Erlangs built in types see this).
Fortunately Erlang comes with a very compact binary type, which I will use. Last time we saw that the output was of the form "..." which indicates strings, Erlang uses a notation of <<...>> to indicate binaries.
Now to get the Port to send binaries to us instead of strings you simply pass, [binary] to sl:start/1 like this sl:start([binary])..

This is normal Erlang behavior for ports and looking at the sl you will see that the start/1 simply calls open_port like this:


start(OpenOpts) ->
%%...
open_port({spawn, "sl_drv"}, OpenOpts).

Now running the previous code with this change results in:

<<"$GPGSV,3,1,12,20,89,000,00,17,45,000,27,11,45,000,33,23,43,000,00*7E\r\n">>
...


So even after specifying that you want to receive binaries, the undocumented mode=line option still breaks messages on linefeed boundaries, nice.

GPS receivers typically uses NMEA messages when reporting position and other information. I use this as my NMEA documentation.

I will start with parsing the GPGSA message and modifies my loop/1 into this:

loop(Port) ->
   receive
       {Port, {data, Data}} ->
           case Data of
               <<"$GPGSA,", Rest/binary>> ->
                   getparams(Rest);
               _ ->
                   io:format("~p~n", [Data])
           end,
           loop(Port);
       stop  ->
           exit(normal);
       {'EXIT', Port, Reason} ->
           exit({port_terminated, Reason})
   end.

The data received from the Port is matched against the pattern <<"$GPGSA,", Rest/binary>> for these messages so I simply call getParams with the rest of the parameters in this case. Next time I will look at getParams and a record representation of the received data.

Reading serial data

The way to communicate with foreign systems from Erlang are through so called Ports. From Joe Armstrongs book the recommended template for doing that goes something like this:

-module(gpsdriver).
-export([start/0, stop/0]).

start() ->
spawn(fun() ->
         register(gps,self()),
         process_flag(trap_exit, true),
         %%openport
         loop(Port)
 end).

stop() ->
gps ! stop.

loop(Port) ->
receive
{Port, {data, Data}} ->
   case Data of
       _ ->
           io:format("Sum: ~p~n", [Data])
   end,
   loop(Port);
stop  ->
   exit(normal);
{'EXIT', Port, Reason} ->
   exit({port_terminated, Reason})
end.

The reason is that the process creating the Port is also the one receiving messages from it.
The code above is slightly simplified, as it only accepts stop messages or Data messages from the port (and process crash messages).

Now to read data from the serial connection, I of course had to read the sl (somewhat sparse) documentation (and the gps device specs). This boils down to communication settings 4800,N,8,1 from the device specs and discovering that it binds to the device /dev/ttyUSB0 on my Linux (remember that tty devices are usually only readable from root, so either do the chmod 666 or run the erl shell as root). From the sl docs a Port is created using start and options set using setopt, so my final start function ends up like (also setting options like receive buffer size, hardware flow control, and an undocumented feature called line mode, which makes the driver break messages on line boundaries):

start() ->
 spawn(fun() ->
               register(gps,self()),
               process_flag(trap_exit, true),
               Port = sl:start(),
               sl:setopt(Port, dev, "/dev/ttyUSB0"),
               sl:setopt(Port, baud, 4800),
               sl:setopt(Port, csize, 8),
               sl:setopt(Port, stopB, 1),
               sl:setopt(Port, parity, 0),
               sl:setopt(Port, hwflow, true),
               sl:setopt(Port, bufsz, 100),
               sl:setopt(Port, mode, line),
               sl:update(Port),
               sl:open(Port),
               loop(Port)
       end).

Remember to put the module in a file with the name of the module and extension .erl, also module names starts with lower-case (upper-case starting names in Erlang are reserved for variables).

Then in the Erlang shell typing:

c(gpsdriver).
gpsdriver:start().

Results in lines like:
"$GPGGA,071852.080,2997.9203,N,03113.4374,E,1,05,2.2,138.8,M,138.8,M,0.0,0000*4D\r\n" "$GPGSA,A,3,17,12,15,28,18,,,,,,,,5.0,2.2,4.5*34\r\n" ...

Stopping the flow using gpsdriver:stop().

Next: parsing time!

Erlang - Serial Blues

So serial comunication does not work out of the box when retrieved from CEAN. This of course boils down to the fact that sl contains a native driver written in C. Now a makefile exists, so I just tried to call make, alas no go! Apparently make includes a file called ../../support/include.mk and this file was no-where in sight, not in any subdirectory of my erlang installation, not in the erlang source distribution, hmm. So this was where luck stroke, trying to peek around for this file I ended up in the Yaws source, and hey there was I file with the right name, so I made a link to that and tried make again... YES succes the C driver builded.

And now doing P = sl:start(). in the erlang shell gave me: #Port<0.100>

Time to do some Erlang ...

Erlang - Install

I of course first needed an Erlang installation. I am running SuSE 10.2 on my main development machine, no Erlang package was available with the distribution, so I had to do a source install.
I downloaded it from erlang.org, the build went very unproblematic, only caveat was that the package pam-devel was not installed. After successfull build Erlang got installed in /usr/local/lib/erlang.

I then went on to install Yaws as I plan to try out some web-development using Erlang as well, this later turned out to be a lucky move.

As my project need to access the serial port (my GPS receiver connects to USB using the serial profile and as such mounts under Linux as a serial device /dev/ttyUSB0), I need a serial library for Erlang luckily not only one but two exists (named sl and serial) I selected the one named sl, solely based on the fact the it is the newer of the two. To get hold of it I decided to use CEAN (Comprehensive Erlang Archive Network) which is Erlang's pendant to Perl's well known CPAN. I managed to get CEAN installed using their FAQ (not without some trouble though).

So with CEAN working, I fired up my erlang shell and typed cean:install(sl). (note the trailing dot). Success! The sl library got installed under /usr/lib/local/erlang/lib/sl-1.0.

After reading the the sl.txt (placed in the doc subfolder under the sl installation folder) to get a clue of how the library worked.
I eagerly did a P = sl:start(). in the erlang shell. only to be met with:
sh: line 0: exec: sl_drv: not found
#Port<0.100>

Alas something was clearly not working!

Next time I will tell how I got this solved and it includes why it was a lucky move to have installed Yaws.

Moving on

OK, I started this blog after I made the decision to try out new programming languages, and also after reading up on a few of them, actually also after deciding which one I will try to investigate further, so I am already far behind on the blogging (not a very good start).

I tried out and read up on:

• Ruby (including Rails)
• Groovy (including Grails)
• Common Lisp (read Common Lisp tutorial for a great tutorial)
• Erlang (I absolutely recommend reading the Programming Erlang )

What I am going to use a lot more time on is Erlang. I have decided on a small pet project to get started.

A while back I bought a GPS receiver and I will try to get position data from this device using Erlang. This is pretty focused and should be relatively easy, now let us see if that holds...

(To be honest I have already started)