Generic Sensor Format (gsf) on github

I'm finally to a point where this is worth talking about.  I started with GSF version 03.06 downloaded from the Leidos website (Leidos split off from SAIC last year).  I used that to start my gsf github repo.  I've given it a really simple GNU Makefile build system and added some basic read testing using Google's gunit/gmock testing suite.  I've setup travis-ci to run the tests every time I push changes.  I even setup Coverity Scan to do static analysis.  I started doing some initial cleanup (spelling, consistently capitalizing GSF in comments), but I need to flush out the unit tests before I start working on the 46 Coverity issues and working on making GSF compile without warnings.  To help with creating tests, I've started writing a pure Python GSF reader so that I can pick packets out of test files and assemble minimally sized test files from real data.  I also need to use the C GSF library to write out some small test cases that cover all the key corner cases.

My original starting point of gsf from Leidos.  It's a shame that they don't have a public repo that I could have cloned with the whole history.

A list of things I'd like to do for GSF:

First green build with Travis-CI:

Coverity summary:

My new ais stream processing code for libais

For a long time, I've had the not so great ais_normalize.py in noaadata to manage multi-line NMEA AIS VDM messages.  It worked for the old USCG format, but it was brittle and cruft code.  Egil added aisdecode to libais based on ais_normalize, but that was building on a terrible foundation.  Today, I pushed the final patch for my nmea_queue module that can handle text, bare NMEA, TAG Block (including Orbcomm extensions), and USCG old CSV metadata.  I still need to add a couple non-VDM messages to the system to work out how to parse those cleanly.  With the new gpsd_format library (led by SkyTruth), libais and gpsd are starting to really play well together.  The design goals of the two are extremely different, so think carefully which you want to use.  In my case, I use both.

Overall, I'm much happier with the state of libais and excited to talk to many folks in the AIS community at the AISSummit 2015 conference in Hamburg next month.  Paul Woods of SkyTruth and I will be giving a pair of talks that will go over how to use "Big Data" techniques and technologies to tackle larger AIS databases with ease.  We will talk about many of the specifics of what has gone into Global Fishing Watch.

I've also been putting work into gpsd, BitVector, and ais-areanotice with a little bit on noaadata (noadata is a mess).  More to come.

Where to put the optional marker in a regex?

I'm looking back at my AIS VDM python regular expressions this morning.  It seems obvious now that I look at this code block, but when a field is missing, I'd rather get a None back, so the "?" goes outside the named regex block.

ipython

In [1]: import re

In [2]: a = re.compile(r'(?P<seq_id>[0-9]?)')

In [3]: b = re.compile(r'(?P<seq_id>[0-9])?')

In [4]: a.match('').groupdict()

Out[4]: {'seq_id': ''}

In [5]: b.match('').groupdict()

Out[5]: {'seq_id': None}

Dynamic Ocean Management

I'm to lazy to switch to a machine that has Photoshop and fix up the images, but check out this article on "Dynamic Ocean Management: Identifying the Critical Ingredients of Dynamic Approaches to Ocean Resource Management" that talks about WhaleALERT!  doi:10.1093/biosci/biv018

The Robert Schwehr Memorial Fund

My dad:

The Robert Schwehr Memorial Fund has just been established at Hidden Villa. It is a beautiful nonprofit educational farm that my dad loved and one of his favorite hiking spots. If you are interested in making a donation, please include the phrase "Robert Schwehr Memorial Fund" in the comment section of the online donation page. Here is the link http://goo.gl/a8CZmi 



http://www.mercurynews.com/peninsula/ci_27585937/los-altos-man-struck-by-car-mountain-view

NOAA CSC and USCG do a lame job with Marine Cadastre AIS

http://marinecadastre.gov/ais/

This is a serious waste of our tax payer dollars.  Public data about vessel location is being removed from datasets.  This stuff is broadcast in the clear for anyone to receive.  And what do they mean by "encrypted"?  Did they just run a hash (e.g. md5) function on it?  If that's the case, if I we can figure the hash function, it's trivial to build a rainbow table style lookup.  Once that's done, there are lots of sources of MMSI to vessel name and call signs.  Oh, and by the way, it's 2015 and only 2009, 2010, and 2011 are available.  And then they publish it in the ESRI File Geo Database (FGDB) format that really isn't open.  And I don't really feel like getting a username and password again (or maybe my old one still works).  Sigh.

"Note: Ship name and call sign fields have been removed, and the MMSI (Maritime Mobile Service Identity) field has been encrypted for the 2010 and 2011 data at the request of the U.S. Coast Guard."

Java and Eclipse on the mac

At one time, Apple seemed into Java and things were only slightly painful.  Then Java went out of favor with Apple and things went from bad to worse.  I downloaded the 64bit Mac build of Eclipse and got this after cd /Applications and tar xf ~/Downloads/eclipse-standard-luna-SR1-macosx-cocoa-x86_64.tar.gz and then double clicking the Eclipse icon.

I tried editing /Applications/eclipse/Eclipse.app/Contents/MacOS/eclipse.ini to look like this:

Looking at my system, I find this for java:

ls -l /Library/Java/JavaVirtualMachines

drwxr-xr-x  3 root  wheel  102 Apr  4  2014 jdk1.7.0_51.jdk

drwxr-xr-x  3 root  wheel  102 May  7  2014 jdk1.7.0_55.jdk

drwxr-xr-x  3 root  wheel  102 Jul  9 13:46 jdk1.7.0_60.jdk

drwxr-xr-x  3 root  wheel  102 Jul 30 13:27 jdk1.7.0_65.jdk

drwxr-xr-x  3 root  wheel  102 Sep 18 05:50 jdk1.7.0_67.jdk

drwxr-xr-x  3 root  wheel  102 Dec 10 08:45 jdk1.7.0_71.jdk

But no luck.  I finally found that this success:

cd /Applications/eclipse/Eclipse.app/Contents/MacOS

./eclipse -vm /Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/bin/

And now I have eclipse running:

So I now have Eclipse Luna Service Release 1 (4.4.1) Build id 20140925-1800 on Mac OSX 10.9.5 with XCode 6.1.1.  It took me way too long to get this figured.