Erlang ssh_scp module

Well I allowed myself time to write a proper and reasonably complete scp client module in Erlang.
It is now released on github as ssh_scp.
This first release allows only to copy from local to a remote location.
So next-up is the other way, receive from a remote.

Erlang client SCP

Okay, I needed to send some data to a unit using Secure Copy (SCP), from an Erlang program.
Well it turned out Erlang did not have a SCP implementation as part of the ssh module, only sftp, but that was not supported by the receiving device, only SCP.

Hmmm, okay I could write the data and use the os:cmd to transfer the file, but where is the fun in that.

So I decided to implement a simple SCP, the mechanics of SCP is described brilliantly here: "How the SCP protocol works".

So here are some code snippets, which shows my take on this in Erlang.

First you need to establish a SSH connection to the receiving side, this is done with the ssh module fro my case like something like this, your connection parameters will vary:
 
 crypto:start(),
 ssh:start(), 
 {ok, ConnectionRef} = ssh:connect("localhost",10022,
   [{user,"root"},
    {user_dir,"/usr/local/etc/ssh"},
    {rsa_pass_phrase,"whatever"}]
   , 30000),

Now you have the ConnectionRef, that is needed. With this you can open a secure channel like this:

 {ok, ChannelId} = 
  ssh_connection:session_channel(ConnectionRef, 30000),

And execute an SCP command on the remote side using this channel:

success = ssh_connection:exec(ConnectionRef, 
                              ChannelId,
                    "scp -tq /tmp", 30000),

The last parameter to the remote scp command is the directory, where your file ends up.
Now I just call:
 wait_for_response(ConnectionRef,ChannelId,FileName,Content,0);

Where FileName parameter is the name of the file ending up at the remote side, and files content is in Content parameter. The wait_for_response implements the SCP protocol like this:

wait_for_response(ConnectionRef,ChannelId,FileName,Content,C) ->
  receive
    {ssh_cm, ConnectionRef, Msg} ->
      case Msg of
        {closed, ChannelId} ->
          ssh_connection:close(ConnectionRef, ChannelId),ok;
        {eof, ChannelId} -> 
          ssh_connection:close(ConnectionRef, ChannelId),
          {error, <<"Error: EOF">>};
        {exit_signal, ChannelId, ExitSignal, 
         ErrorMsg, _LanguageString} ->
          ssh_connection:close(ConnectionRef, ChannelId),
          {error, list_to_binary(io_lib:format(
                    "Remote SCP exit signal: ~p : ~p", 
                    [ExitSignal,ErrorMsg]))};
        {exit_status,ChannelId,ExitStatus} ->
          ssh_connection:close(ConnectionRef, ChannelId),
          case ExitStatus of
            0 -> ok;
            _ -> {error,
                  list_to_binary(io_lib:format(
                     "Remote SCP exit status: ~p",[ExitStatus]))}
          end;
        {data,ChannelId,_Type,<<0>>} ->
          SendResponse = 
          case C of 
            0 -> %%ok send header
              Header = 
                list_to_binary(lists:flatten(
                  io_lib:format("~s ~p ~s~n",
                  ["C0644", size(Content), FileName]))),
                ssh_connection:send(ConnectionRef,
                                    ChannelId, 
                                    Header);
            1 -> %%ok send file
              ssh_connection:send(ConnectionRef, 
                                  ChannelId, 
                                  Content),
              ssh_connection:send(ConnectionRef, 
                                  ChannelId,
                                  <<"\0">>);
            2 -> %%ok file transferred
              ssh_connection:send_eof(ConnectionRef, ChannelId)
          end,
          case SendResponse of
            ok -> %%recurse
              wait_for_response(ConnectionRef,
                                ChannelId,
                                FileName,
                                Content,C+1);
            _ -> SendResponse
          end
      end
  end.  


Notice that all files are ends up with permission 0644 as per the Header:
io_lib:format("~s ~p ~s~n",["C0644", size(Content), FileName])
and how the parameter C holds the protocol state.

And don't for get to close your connection when done:  ssh:close(ConnectionRef)

I might someday get the time to write a feature complete SCP module for Erlang, But till then, these snippets might help some of you.

Erjang

Just took Erjang (www.erjang.org) for a spin.

I ran the ring_test example and above is a simple plot of performance compared with Erlang.

First is java with -server option second java, then erlang and finally erlang with hipe.

Note that in this case hipe is actually slower, and java with server option is also slower.

I think it is amazing that Erjang is only ~30% slower, great work

Mnesia

Long christmas :-)

Been busy doing my day job as a Java consultant, and topped it by volunteering to do an Mnesia introduction at the first (hopefully not last) Erlang meetup in Århus. It was held at Lenio and it was very nice to meet local erlangers.

My presentation can be seen here

Client process

First a little catching up.
Time and date parsing, I decided to use the definition from Erlangs calendar module which are:

date() = {Year, Month, Day}
  Year = int()
  Month = 1..12
  Day = 1..31

time() = {Hour, Minute, Second}
  Hour = 0..23
  Minute = Second = 0..59

Only I need millisecond precision for the time so i decided on this compatible format
{{Hour, Minute, Second}, Millis}

Code for generating these is easy using iolib:fread/2

    {ok,[Day,Month,Year],[]} = io_lib:fread("~2d~2d~2d",Date),
    {ok,[Hour,Minute,Second,Millis],[]} =io_lib:fread("~2d~2d~2d.~3d",Time),

I get a list of the parsed values which can be passed directly into the relevant tuples.

     {fixtime,{{Hour,Minute,Second},Millis}},
     {date,{Year+2000,Month,Day}}

Since last I noticed that the GPS device prior to getting a satellite fix returned empty lists in places where the code expected integers, floats or atoms, leading to errors in the call to list_to_integer for example, so I had to replace these with functions like these:

 safe_to_float([]) ->
    -1.0;
 safe_to_float(F) -> 
    list_to_float(F).

Now to the client process. I simple spawn a new process which will receieve messages from the driver and react only to gga messages as they are carrying the position information, then they format these in a google maps friendly way. The entire code goes like this:

start() ->
    spawn(fun() ->
                  gpsdriver:start(self()),
                  loop()
          end).
loop() ->
    receive
        {gga,
         {fixtime,_},
         {lattitude,[Lattitude,NS]},
         {longitude,[Longitude,EW]},
         {quality,_},
         {numberofsatellites,_},
         {horizdi,_},
         {altitude,_},
         {geoidheight,_},
         {timesincedgps,_},
         {dgpsstation,_}} ->
            io:format("~p~s,~p~s~n",[Lattitude,NS,Longitude,EW]);
        _ ->  true
    end,
    loop().

Note that in the start/0 function I pass the Pid (process id) of the newly created process to the gpsdriver, so that was changed to accept this and instead of outputting directly to standard output sends the tuples to this process using Erlang message sending mechanism Pid ! Record

The final code outputs messages like:

29.9792N,31.1343E
29.9792N,31.1343E
29.9792N,31.1343E


So this completes the gps project for me, only that I will make some performance measures to get an idea of the best way to parse the incoming messages.

However this has to be postponed to the new year, as a major holiday is approaching fast.

Best wishes for 2008 to everyone who bothers reading this.

Parsing take #3

I want to make more structured messages so I parse the lists using a function I call compose/1 for this I again use Erlangs superior pattern matching.

So for GPGSA the compose looks like:

compose(["GPGSA",A,B,C1,C2,C3,C4,C5,C6,C7,C8,C9,C10,C11,C12,D,E,F]) ->
    {gsa,
     {auto,list_to_atom(A)},
     {fix,list_to_integer(B)},
     {satellites,satellites([C1,C2,C3,C4,C5,C6,C7,C8,C9,C10,C11,C12],[])},
     {pdop,list_to_float(D)},
     {horizdi,list_to_float(E)},
     {vertdi,list_to_float(F)}};

This will only match the lists with "GPGSA" sentences (my GPS device only submits the sentences GPGSA,GPGGA,GPRMC,GPGSV).

The structured data I use are simple tuples, where I use Erlang atoms as identifiers for the fields, as you can see I also translate the integer data into integers (and for other messages to float values). A helper function is used so I do not repeat my self to much:

satellites([[] | _], Satellites) ->
   Satellites;
satellites([A | Rest], Satellites) ->
   satellites(Rest, [list_to_integer(A) | Satellites]).

The rest of compose for parsing the remaining sentences looks like:

compose(["GPGGA",A,B1,B2,C1,C2,D,E,F,G1,G2,H1,H2,I,J]) ->
   {gga,
    {fixtime,A},
    {lattitude,[list_to_float(B1)/100,list_to_atom(B2)]},
    {longitude,[list_to_float(C1)/100,list_to_atom(C2)]},
    {quality,list_to_integer(D)},
    {numberofsatellites,list_to_integer(E)},
    {horizdi,list_to_float(F)},
    {altitude,[list_to_float(G1),list_to_atom(G2)]},
    {geoidheight,[list_to_float(H1),list_to_atom(H2)]},
    {timesincedgps,list_to_float(I)},
    {dgpsstation,J}};

compose(["GPRMC",A,B,C1,C2,D1,D2,E,F,G,H1,H2]) ->
   {rmc,
    {fixtime,A},
    {status,list_to_atom(B)},
    {lattitude,[list_to_float(C1)/100,list_to_atom(C2)]},
    {longitude,[list_to_float(D1)/100,list_to_atom(D2)]},
    {speed,list_to_float(E)},
    {trackangle,list_to_float(F)},
    {date,G},
    {magneticvariation,[H1,H2]}};

compose(["GPGSV",A,B,C | SateliteData ]) ->
   {gsv,
    {numberofsentences,list_to_integer(A)},
    {number,list_to_integer(B)},
    {numberofsatellites,list_to_integer(C)},
    satellitedata(SateliteData, [])}.

I divide the longitude/lattitudes by 100 as I wish that they closely match the expected format for Google maps.

The GPGSV sentences are variable length so I use a helper function for parsing those:

satellitedata([D,E,F,G | SateliteData ], Satellites) ->
   satellitedata(SateliteData, 
                 [{sattelite,list_to_integer(D)},
                  {elevation,list_to_integer(E)},
                  {azimuth,list_to_integer(F)},
                  {snr,list_to_integer(G)} | Satellites]);
satellitedata([], Satellites) ->
   Satellites.

My final loop/1 looks like this:

loop(Port) ->
   receive
       {Port, {data, Data}} ->
           case Data of
               <<"$", Rest/binary>> ->
                   Tokens = tokenize (Rest),
                   Record = compose(Tokens),
                   io:format("~p~n", [Record]);
               _ ->
                   sl:setopt(Port, mode, line),
                   sl:update(Port)
           end,
           loop(Port);
       stop  ->
           exit(normal);
       {'EXIT', Port, Reason} ->
           exit({port_terminated, Reason})
   end.

The resulting messages are reported in the shell like:

{rmc,{fixtime,"214612.620"},
    {status,'A'},
    {lattitude,[29.9792,'N']},
    {longitude,[31.1343,'E']},
    {speed,9.00000e-2},
    {trackangle,81.9100},
    {date,"291107"},
    {magneticvariation,[[],[]]}}
{gga,{fixtime,"214613.620"},
    {lattitude,[29.9792,'N']},
    {longitude,[31.1343,'E']},
    {quality,1},
    {numberofsatellites,4},
    {horizdi,4.00000},
    {altitude,[146.2000,'M']},
    {geoidheight,[0.0000,'M']},
    {timesincedgps,0.00000e+0},
    {dgpsstation,"0000"}}
{gsa,{auto,'A'},
    {fix,3},
    {satellites,[2,23,20,31]},
    {pdop,7.30000},
    {horizdi,4.00000},
    {vertdi,6.00000}}
{rmc,{fixtime,"214613.620"},
    {status,'A'},
    {lattitude,[56.2778,'N']},
    {longitude,[10.2091,'E']},
    {speed,0.100000},
    {trackangle,100.610},
    {date,"291107"},
    {magneticvariation,[[],[]]}}

Note I still need to parse the dates and times into some nice representations.
I will next time make an additional process which will receive these messages and report only the position data in a format compatible with Google maps

Parsing take #2

NMEA messages are quite simple. They are simply comma separated values (some values might be absent) with start/end markers and a checksum.
The checksum is the xor of the bytes between the start and end markers, and is sent as a hex encoded value after the end marker. Like this:


$GPGSA,A,3,11,20,01,17,,,,,,,,,7.0,2.4,6.6*36\r\n


$ being the start marker * being the end marker and 36 hex being the checksum.

Last time I made the decision to match on the start of the message like:

           case Data of
               <<"$GPGSA,", Rest/binary>> ->
                   getparams(Rest);
               _ ->
                   io:format("~p~n", [Data])
           end,

This was actually not a good decission, because the NMEA messages are so alike I find it better to simply match on the start marker and receive a list of parsed values (tokens).

So my new case looks like this:

            case Data of
                <<"$", Rest/binary>> ->
                    Tokens = tokenize (Rest),
                    io:format("~p~n", [Tokens]);
                _ ->
                    sl:setopt(Port, mode, line),
                    sl:update(Port)
            end,

Now the second _ clause should never match, but when you call the stop/0 and then start/0 funktions the sl module sometimes (read bug) looses the information that line mode was requested, in these cases this clause will match and I force the line mode on the Port, which works around this.

Now to split the received binary into tokens I call tokenize/1, now I was hoping to keep the data in binaries and build a list of these, but constructing binaries by continuously appending bytes to them is not supported in erlang, so for a first go I will turn the binary into a list of strings (a possible solution to my first wish is to scan the binary, record the comma positions and use the split_binary built in function, I will try this later and compare the performance implications).

Now the tokenize will have two missions: tokenize the binary and verify the checksum

First tokenize/1:

tokenize(<<$G:8, Rest/binary>>) ->
    tokenize(Rest, [$G], [], $G).

Here I use that all NMEA messages starts with "G", so I can initialize the current token (second parameter) with a $G ($ is Erlangs syntax for characters) and the calculated checksum to the same value (fourth parameter) and the third parameter is the current list of parsed tokens which start out empty.

Now tokenize/4:

tokenize(<<$,:8 , Rest/binary>>, Token, Tokens, Sum) ->
    tokenize (Rest, [], [lists:reverse(Token) | Tokens], Sum bxor $,);
tokenize(<<$*:8 , Rest/binary>>, Token, Tokens, Sum) ->
    checksum (Rest, [lists:reverse(Token) | Tokens], [], Sum);
tokenize(<>, Token, Tokens, Sum) ->
    tokenize(Rest, [N | Token], Tokens, Sum bxor N).

The algorithm is simple when a comma is first in the binary, a token is finished so it is added to the list of parsed tokens (as I add characters at the beginning of the list in usual Lisp style, the lists:reverse needs to be called) and $, is added to the checksum. When a * is first we move on to parse the checksum, in all other cases the first byte in the binary is added to the current token and the checksum.

Now the checksum needs to be parsed and verified, the code below does this, note that the first clause with the empty binary should in theory never match, but again in some circumstances (read stop/start again) this can happen and this takes care of that. Otherwise it is pretty straight forward, the read checksum is converted to its numerical value using {_,ReadSum,_} = io_lib:fread("~16u",Chk),:

checksum(<<>>, _, _, _) ->
    {error};
checksum(<<$\r:8, Rest/binary>>, Tokens, Chk, Sum) ->
    checksum(Rest, Tokens, lists:reverse(Chk), Sum);
checksum(<<$\n:8>>, Tokens, Chk, Sum) ->
    {_,ReadSum,_} = io_lib:fread("~16u",Chk),
    case lists:nth(1,ReadSum) of
        Sum -> lists:reverse(Tokens);
        _ -> {error}
    end;
checksum(<>, Tokens, Chk, Sum) ->
    checksum(Rest, Tokens, [N | Chk ], Sum).

Thats it, we get the binary split into a list of tokens, next I will turn these lists into more structured records.