Sunday, March 3, 2019

Martian dust / soil

Back in 1990, I had just graduated from high school and was working for Bill Borucki at NASA Ames.  I wrote a report on Martian Dust and electrical charge/discharge.  I remember working so hard on that report trying to figure out how to do background research and digging through the NASA libraries.  For all that hard work, I didn't end up with much.  My report is painful to read now.  It's full of typos and doesn't actually say much.  Now we have wikipedia and there are these:


And so on and so forth...

Grains in desert sandstorms spontaneously generate strong electrical charges; likewise volcanic dust plumes produce spectacular lightning displays. Charged particle clouds also cause devastating explosions in food, drug and coal processing industries. Despite the wide-ranging importance of granular charging in both nature and industry, even the simplest aspects of its causes remain elusive, because it is difficult to understand how inert grains in contact with little more than other inert grains can generate the large charges observed. Here, we present a simple yet predictive explanation for the charging of granular materials in collisional flows. We argue from very basic considerations that charge transfer can be expected in collisions of identical dielectric grains in the presence of an electric field, and we confirm the model's predictions using discrete-element simulations and a tabletop granular experiment.

Tuesday, January 22, 2019

If I were to redo my 2011 Research Tools class

If I were doing the equivalent of my 2011 Research Tools class, I would do it as much as possible in Python 3 (python 2 is dead) with Jupyter notebooks.  I don't know what editor, but maybe PyCharms?  But I would prefer to pick something opensource.  Here are some of the things I was just thinking about while on the bus this morning:


  • Teaching python should be taught through multiple methods.  Incorporate or at least give guided options for self learning, normal classes focused on using Python and Jupyter, included in other courses that the same students will take.
  • Teach unit testing from the beginning
  • Beware the Python examples in some commercial products.  Their examples can sometimes be kinda crazy and detrimental to learning how to write Python effectively.
  • Start with version control from the beginning for projects of all sizes.  There is no project too small for git (or I guess mercurial)
  • Treat python 2 as persona non grata
  • Introduce style guides early.  If you can, pick one for the department.  Code should be like formal writing and lab notebooks: Correct grammar, spelling, formatting, etc. make things grow faster and last longer. e.g. http://google.github.io/styleguide/pyguide.html
  • To go with the style guide, teaching about code formatting tools (and having a required one) can take away a lot of stress from the code.  Go ahead and write that messy looking code.  Then run the formatter and linter when you are ready.  A much lower stress way of getting readable code.  codespell is another of my favorite tools
  • Teach pip + virtual environments
  • Teach the basics of copyright rules and what licenses mean.  When is it okay to use other code.  When is it okay or not okay to share your code with others.  What does it mean to not put a license on code
  • Encourage sending pull requests to projects.  Especially for projects with CI setup so you can see what it's like to have your code checked
  • Stack overflow and the like are used by the pros.  Everyone should use them and learn how to interpret the quality of answers

Monday, July 23, 2018

GeoTools Hello World on the Mac

Last time I did a hello world on Linux (Debian Testing).  This time, I'm going to go through trying it on the Mac.  If you use home brew, mac ports, or something else, you may see something different.  Also, my install of Java 9 on my mac isn't likely 100% normal.  I'm too new to know the differences.

The resulting pom.xml and App.java are exactly the same as with my prior post, which is the way it is supposed to be!

https://gist.github.com/schwehr/b433076cab0cdb99c61163c6fb56bf7f

Getting setup with JAVA_HOME is where I hit some speed bumps.  But they turned out not to be too bad.  So I know this is where I want to end up pointing my JAVA_HOME:

/Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home

But first, I wanted to see what else is around.  Here is my default path:

type javac
javac is /usr/bin/javac

javac --version
javac 9

java --version
java 9
Java(TM) SE Runtime Environment (build 9+181)
Java HotSpot(TM) 64-Bit Server VM (build 9+181, mixed mode)

I wanted to see if I could use the default /usr/bin stuff.  Before I try that, I need to install maven.  I use fink, but I bet brew and macports have it too.

fink install maven

mvn --version
Error: JAVA_HOME is not defined correctly.
  We cannot execute /usr/libexec/java_home/bin/java

Uh oh!  Time to see what /usr has.

ls -ld /usr/libexec/java_home 
lrwxr-xr-x  1 root  wheel  79 Jan 31 14:36 /usr/libexec/java_home -> /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java_home

find /System/Library/Frameworks/JavaVM.framework/Versions -name javac
/System/Library/Frameworks/JavaVM.framework/Versions/A/Commands/javac

That doesn't look like the correct structure, so I'm going to avoid that and go with what I originally thought would work:

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home

mvn --version
Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; 2014-08-11T13:58:10-07:00)
Maven home: /sw/share/maven
Java version: 9, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"

So now it looks like I've got maven and a probably working Java env.  Time to start building the app.  Start by creating a space to work on it and create the initial project:

cd && mkdir maven && cd maven

mvn --batch-mode archetype:generate -DgroupId=com.example -DartifactId=geotools-hello -DarchetypeGroupId=org.apache.maven.archetypes

Now checkout what's in the default setup:

cd geotools-hello/

tree -d

find . -type f
./pom.xml
./src/test/java/com/example/AppTest.java
./src/main/java/com/example/App.java

Can I build anything with it?

mvn compile

[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] Source option 1.5 is no longer supported. Use 1.6 or later.
[ERROR] Target option 1.5 is no longer supported. Use 1.6 or later.
[INFO] 2 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

Nope!  This is the same issue I had on Linux.  Time to start editing the pom.xml file.  Add this:

  <properties>
    <maven.compiler.source>1.9</maven.compiler.source>
    <maven.compiler.target>1.9</maven.compiler.target>
  </properties>

Now is is actually able to build and run the application!  I added a line to print out the Java version.  On the Mac, I'm seeing "9", while on Linux I was seeing 9.0.4.

mvn compile exec:java -Dexec.mainClass="com.example.App"
[INFO] Scanning for projects...
Downloading: https://repo.maven.apache.org/maven2/org/codehaus/mojo/exec-maven-plugin/maven-metadata.xml
[SNIP]
Downloaded: https://repo.maven.apache.org/maven2/org/codehaus/mojo/exec-maven-plugin/1.6.0/exec-maven-plugin-1.6.0.jar (57 KB at 434.0 KB/sec)
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building geotools-hello 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ geotools-hello ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /Users/schwehr/maven/geotools-hello/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ geotools-hello ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /Users/schwehr/maven/geotools-hello/target/classes
[INFO] 
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ geotools-hello ---
Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.2.1/maven-reporting-api-2.2.1.pom
Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting-api/2.2.1/maven-reporting-api-2.2.1.pom (2 KB at 15.1 KB/sec)
Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/reporting/maven-reporting/2.2.1/maven-reporting-2.2.1.pom

[SNIP]

Downloaded: https://repo.maven.apache.org/maven2/commons-cli/commons-cli/1.2/commons-cli-1.2.jar (41 KB at 161.3 KB/sec)
Downloaded: https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.20/plexus-utils-3.0.20.jar (238 KB at 730.6 KB/sec)
Hello World!
9

Now I change App.java to include.  It works!

mvn compile exec:java -Dexec.mainClass="com.example.App"
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building geotools-hello 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ geotools-hello ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /Users/schwehr/maven/geotools-hello/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ geotools-hello ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /Users/schwehr/maven/geotools-hello/target/classes
[INFO] 
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ geotools-hello ---
Hello World!
9
GeoTools version 20-SNAPSHOT (built from rc52833cadf758699a1969d258ab6172b305f9416)
Java version: 9
Operating system: Mac OS X 10.13.6
GeoTools jars on classpath:

Tuesday, July 3, 2018

GeoTools hello world

I just wanted to get started playing with GeoTools.  When you are a beginner to Java, Maven, and GeoTools, it's impressive how many pitfalls there are.  I did just about everything wrong and found all sorts of dark alleys that involved my machine downloading junk for a hour and then declaring failure.  I can't believe how much time it took me to figure out how to get this to work on my Linux workstation.

Now that I have it working, it's time to document what I did.  My environment is pretty close to Debian Testing.  You will probably find things in this description that don't work for you.  I've redacted a few little things here and there, so expect the output you see to be much more verbose.

App.java and pom.xml as I have them at the end of this are here:

https://gist.github.com/schwehr/b433076cab0cdb99c61163c6fb56bf7f

The first hurdle is installing Java and the JDK.  I've got Java 9 preinstalled on my workstation in a funky location, so pardon that confusion.  Java lives here for me:

ls /usr/local/buildtools/java

The first thing I need to do is be explicit about which version of Java to use.  My system is setup to default to Java 8, and why not live from head?

export JAVA_HOME=/usr/local/buildtools/java/jdk9

$JAVA_HOME/bin/java -version
openjdk version "9.0.4"
OpenJDK Runtime Environment (build 9.0.4+11)
OpenJDK 64-Bit Server VM (build 9.0.4+11, mixed mode)

Now I need Apache Maven to control the build and run process.

apt-cache show maven | grep Ver
Version: 3.5.0-5

sudo apt-get install maven

mvn -version
Apache Maven 3.5.0
Maven home: /usr/share/maven
Java version: 9.0.4
Java home: /usr/local/buildtools/java/jdk9
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.9.0-6-amd64", arch: "amd64", family: "unix"

Where is maven?

ls -l /usr/bin/mvn
lrwxrwxrwx 1 root root 21 May 24 21:56 /usr/bin/mvn -> /etc/alternatives/mvn

readlink -f /usr/bin/mvn
/usr/share/maven/bin/mvn

dpkg -S `readlink -f /usr/bin/mvn`
maven: /usr/share/maven/bin/mvn

Let's start a project that will be our starter.

mkdir maven && cd maven

mvn --batch-mode archetype:generate -DgroupId=com.example -DartifactId=geotools-hello -DarchetypeGroupId=org.apache.maven.archetypes

tree -d

cd geotools-hello

find .
.
./pom.xml
./src
./src/main
./src/main/java
./src/main/java/com
./src/main/java/com/example
./src/main/java/com/example/App.java
./src/test
./src/test/java
./src/test/java/com
./src/test/java/com/example
./src/test/java/com/example/AppTest.java

mvn compile

[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] Source option 1.5 is no longer supported. Use 1.6 or later.
[ERROR] Target option 1.5 is no longer supported. Use 1.6 or later.

Well, that sucks!  I've got Java 9 and this thing is trying to use 1.5?  According to Java version history, 2004 is calling and they want their old Java back.  Edit the pom.xml and tell it to get to at least the year 2017.

  <properties>
    <maven.compiler.source>1.9</maven.compiler.source>
    <maven.compiler.target>1.9</maven.compiler.target>
  </properties>

Try it again and it should work.

mvn compile

mvn exec:java -Dexec.mainClass="com.example.App"
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building geotools-hello 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ geotools-hello ---
Hello World!

Now we want to make sure the java version is what we expect.  Edit src/main/java/com/example/App.java to look like this:

package com.example;

import java.lang.System;

public class App
{
    public static void main( String[] args )
    {
        System.out.println( "Hello World!" );
        System.out.println(System.getProperty("java.version"));
    }
}

Running it should now give a little bit more info.  We can compile and run in one maven command.  The commands added will be run in the order given on the command line.

mvn compile exec:java -Dexec.mainClass="com.example.App"
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building geotools-hello 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ geotools-hello ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /usr/local/google/home/schwehr/maven/geotools-hello/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.6.1:compile (default-compile) @ geotools-hello ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /usr/local/google/home/schwehr/maven/geotools-hello/target/classes
[INFO] 
[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ geotools-hello ---
Hello World!
9.0.4

So we indeed have java 9.  Woot!

Now on to trying to do the simplest possible thing I can think of with GeoTools.  Can we print the version information?  Finding the GeoTools.getAboutInfo() method took me way to long.  Sigh.

Edit the App.java file to import org.geotools.factory.GeoTools and call the static method:

package com.example;

import java.lang.System;
import org.geotools.factory.GeoTools;

public class App
{
    public static void main( String[] args )
    {
        System.out.println( "Hello World!" );
        System.out.println(System.getProperty("java.version"));
        System.out.println(GeoTools.getAboutInfo());
    }
}

Next you need to make your pom.xml match mine.  You need to add the geotools version to the properties.  Then you need to add the gt-metadata library to the dependencies.    And then add the repositories where the dependencies can be found.

  <properties>
    <maven.compiler.source>1.9</maven.compiler.source>
    <maven.compiler.target>1.9</maven.compiler.target>

    <geotools.version>20-SNAPSHOT</geotools.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.geotools</groupId>
      <artifactId>gt-metadata</artifactId>
      <version>${geotools.version}</version>
    </dependency>
  </dependencies>

  <repositories>
    <repository>
      <id>maven2-repository.dev.java.net</id>
      <name>Java.net repository</name>
      <url>http://download.java.net/maven/2</url>
    </repository>
    <repository>
      <id>osgeo</id>
      <name>Open Source Geospatial Foundation Repository</name>
      <url>http://download.osgeo.org/webdav/geotools/</url>
    </repository>
    <repository>
      <snapshots>
        <enabled>true</enabled>
      </snapshots>
      <id>boundless</id>
      <name>Boundless Maven Repository</name>
      <url>http://repo.boundlessgeo.com/main</url>
    </repository>
  </repositories>

Now try it and hopefully you will succeed with the about showing you a couple of lines about GeoTools.

mvn compile exec:java -Dexec.mainClass="com.example.App"

Hello World!
9.0.4
GeoTools version 20-SNAPSHOT (built from r9accd1de3af6dcbd69ee56eebd4375a5b5ceb19b)
Java version: 9.0.4
Operating system: Linux 4.9.0-6-amd64
GeoTools jars on classpath:

Finding that GeoTools.getAboutInfo() was in gt-metadata was a bit painful.  I wandered through the upstream source's pom.xml files and got very confused.  In the end, I confirmed the location after thrashing about and having maven download a huge amount of stuff to ~/.m2 (maven's cache directory) and inspecting the compiled jars.  Silliness ensued...

cd ~/.m2

find . -name \*.jar | while read f ; do (jar tf $f | awk '{print "'"$f"'" " " $0}' | grep -i "GeoTools") ; done

jar tf ./repository/org/geotools/gt-metadata/20-SNAPSHOT/gt-metadata-20-SNAPSHOT.jar | grep factory/GeoTools.class
org/geotools/factory/GeoTools.class

mkdir foo

cp ./repository/org/geotools/gt-metadata/20-SNAPSHOT/gt-metadata-20-SNAPSHOT.jar ~/foo

cd ~/foo

jar xf gt-metadata-20-SNAPSHOT.jar

find . -name GeoTools.class
./org/geotools/factory/GeoTools.class

strings org/geotools/factory/GeoTools.class | grep About
getAboutInfo

Now to do stuff with GeoTools that is more interesting than just a hello world.

Monday, May 21, 2018

What could the HDFGroup do to improve HDF5 (and HDF4)

Back in 2014, I tried to do a bit of cleanup on HDF4 & HDF5.  I'm sure my comments are long out of date...


A few things would accelerate contributions from the community:

  • Mirror the git repos to github and set them up to automatically get updated
  • Setup continuous integration (VI) with travis-ci, appveyor, coveralls, and/or other providers
  • Allow bugs and pull requests on github.  Then people can propose patches and make sure they don't break the builds
  • Make sure primary development is going towards the "master" branch.  That's were people expect to look for the latest changes with git based code.  That's what people expect to send you patches against.

Even and I wrote down some of what we do for GDAL, but it's getting out of date after a few years.  Use these and any other tool you can (e.g. clang-tidy modernize).

http://erouault.blogspot.com/2016/01/software-quality-improvements-in-gdal.html

Other things that I personally think would be great for the HDF team to do:

  • Sign up for OSS Fuzz.  Use google resources try to crash the code and send you bug reports
  • Change the tests to only write into a user controllable temp location.  I run tests from a CAS filesystem that is read only and the runner gets passed the location of where it is allowed to write.
  • Define what the acronyms are in files.  I don't remember what the "AC" means even when I'm in H5AC.c.
  • Drop things like who created each file and when.  Those things are in the version control history and are noise when debugging
  • Have more tests that try to do things in isolation - true unit tests.  e.g. just test a single function.  I usually expect that testing code should be as long or longer than each file.  And each test case only tests one thing
  • Make sure that the code becomes and stays whitespace clean.  Best is to use clang-format, but things like perl -pi -e 's/\s+\n/\n/g' work quite well
  • Don't use line feed characters in the source.  That makes viewing on large screens harder with some tools
  • Consider using a test framework like googletest/gunit.  Many people are experienced with these frameworks and can better follow your tests and contribute back
  • Switch to C++11 as the minimum version of C++.  So many things get easier/better e.g. https://trac.osgeo.org/gdal/wiki/rfc68_cplusplus11

Monday, May 7, 2018

Paywalled docs

"Maritime navigation and radiocommunication equipment and systems - Digital interfaces - Part 460: Multiple talkers and multiple listeners - Ethernet interconnection - Safety and security"

This doc is out, but I have never read it. If the maritine community wants this stuff to really work, you need to be engaging security experts.  None of us that work on computer security are going to pay for this doc or any of it's siblings. It is time for the IEC to release all AIS/eNav standards as free docs. You need to get them in as many hands as possible if you want to get towards better systems.   Not just the big hardware and software providers...

Monday, September 25, 2017

GPS spoofing possibly seen in AIS data

Disclaimer:  The opinions here are my own and do not represent Google, SkyTruth, Global Fishing Watch, or the Univ. of New Hampshire.  I work for Google, co-founded Global Fishing Watch, and am an Affiliate Professor at the Univ. of New Hampshire.

https://twitter.com/schneierblog/status/912307960369504259

Warning:  This was written off the top of my head in less than an hour with no review and not much fact checking.  I know I'm missing lots of interesting details.

Before I go an read the article that Bruce Schneier linked to, I'll try to think through what could explain these general sorts of issues.  I've heard rumbling before of this in AIS data, but it's time to take a few minutes.

After trawling through AIS data from recent years, evidence of spoofing becomes clear. Goward says GPS data has placed ships at three different airports and there have been other interesting anomalies. "We would find very large oil tankers who could travel at the maximum speed at 15 knots," says Goward, who was formerly director for Marine Transportation Systems at the US Coast Guard. "Their AIS, which is powered by GPS, would be saying they had sped up to 60 to 65 knots for an hour and then suddenly stopped. They had done that several times."
All of the evidence from the Black Sea points towards a co-ordinated attempt to disrupt GPS. A recently published report from NRK found that 24 vessels appeared at Gelendzhik airport around the same time as the Atria. When contacted, a US Coast Guard representative refused to comment on the incident, saying any GPS disruption that warranted further investigation would be passed onto the Department of Defence.
So what could cause deviations in speed?  This is without looking deeper at the data.   First, I need to think about what is being recorded here.  A ship has an AIS device that is trying to broadcast it's position.  It has an internal sense of location.  Where does that come from?  It usually has a GPS receiver, but that isn't always where the position comes from.  It could be from a different GNSS (aka satellite) nav system, Loran (Loran is not dead - only the US version), dead reckoning, or other ways of entering position.

If it was a GPS, there could be a couple things go on.  The GPS might be receiving differential corrections or ephemeris from someplace other than the satellites.  That could be a direct RF link or some internet link.  Those could be bogus either intensionally or unintentionally.  The GPS could be defective and I've seen that very clearly before.

But it might not be GPS.  These ships could be using pure GLONASSGalileo, or other system.  Or it could be a receiver that blends multiple constellations.  If one of these other systems has a problem, I'm not sure how that would impact the resulting position and velocity fixes.

The USCG and Europeans stopped using LORAN, but that doesn't mean there are no places where the same or similar local ground systems are in use for positioning.  The Russians have CHAYKA.  I've got used them and haven't thought through their failure modes or how to attack them.  They have a long and interesting history with things like Decca.

We also have the possibility the ship operator modifying the position.   Global Fishing Watch has seen clear evidence of ships monkeying with the position values on their AIS transceivers to thwart observers.  That could be from a modified AIS unit (e.g. customized embedded software), modified GPS, or a device between the GPS hardware and the AIS hardware.  A simple way to do this is to just use a computer with a software defined radio and send what ever you want.

There may also be Inertial Navigation System (INS) between any positioning system(s) and the AIS's embedded computer.  What are the ways an INS can have trouble?  There are plenty.

But, the ship's AIS might not have even been on a position system.  If the system's connection to a position system fails or it thinks it failed, the system can dead reckon.  Depending on the sensor inputs available, you can get all sorts of fun results.  And if the system later gets positioning input back, how does if get itself back in line?  Does it jump to the most recent position or does it try to gradually catch up.

There is probably plenty more to positioning, but it is time to move on to other topics like speed and time.

Next I wonder what is being used for speed.  AIS messages 1-3 have a speed (not velocity) field.  It's up to the AIS unit to fill that out.  How is it doing that?  It would be easy to alter or have that wrong.  I'm thinking of the NOAA Ship Thomas Jefferson that had bad bits in their AIS messages for months after a lightning strike to the ship.  Or are people with the messages using the times of the messages and their locations to externally calculate a speed for the ship?  That begs the question of what time to associate with each message and more specifically the position within the message.

Each AIS message 1,2,3 reports the seconds within the minute of the position report.  Only some reports have a more detailed time record with hour and minute (in the comm state block).  So if you are looking at a message, there are several questions.  Did the system know what time it was?  How far off could the time be?  Does the system think it doesn't have a decent sense of time from the GPS and internal clock(s)?  A substantial number of AIS transceivers report that they are getting time from the AIS transceivers around them.  What does that mean when most systems have a built in GPS that should provide an accurate sense of time?

Then we have the question of how long from the position/time tuple does it take the AIS to form the AIS message and get it out of the RF link?  It should be quick (less than 3-4 seconds), but who knows what a misbehaving transceiver might do.  And there is the added factor that AIS units will listen and act on remote commands without any authentication at all.  You can even tell an AIS unit to switch frequency.  That was accidentally demonstrated in the wild by the USCG of the US East coast many years ago.  Whoops!

So now we have an AIS message out on RF.  Where did it come from?  There is nothing in an AIS message that proves it is from that ship.  There is no cryptographic signature or other trick to make sure an external source isn't providing position.  You could jam the slots the ship transmits on and make fake messages for the ship in other slots and it would be very hard to spot from just typical NMEA logs of AIS.

There are plenty of cases with multiple ships having the same MMSI.  That's not a huge deal when they are in different oceans, but what happens when they are near each other?

Next, some AIS receivers do a careful job of recording metadata for a received AIS message, but most do not.  Some don't even record a received timestamp.  So you have to use the overall NMEA stream to keep a sense of time in the logs.  The USCG put atomic clocks in their receiving system (woo hoo, must be perfect time logs...), but then suffered Windows Time Protocol issues that had their logging systems recording times up to 24 hours different from the actual receive time.  Some people just use the loggers internal clock without NTP... that's full of all sorts of fun.  Sadly, oscillators in computers in are also temperature sensors.  They alter their frequency some with temperature.  I've seen that with a logger on the end of a peer in New Hampshire and it was especially bad when the 24 hour temperature swing was as large as near 0F to 60F.

There is also the case of where is the timestamp being recorded.  Back in 2010, I was getting Orbcomm data via the USCG.  About every 6 hours, I'd get a huge dump of AIS messages.  The timestamp metadata for all those messages would be the time of the dump.  So not helpful.

Then once a message is "received" there is method of getting the data to the decoder.  As seen by many, you might not be getting what you think.  If someone has access to the network, they could inject fake / altered messages and metadata.  If you can't trust the network or the people in the network, who knows what you will get.  And even in a trusted situation, you may not be receiving what the receiver saw.  The USCG provides NAIS feeds that decimate the data to things like only 1 copy of a message received at multiple points in the network and only 1 message per minute from a particular MMSI.  How do you check such decimated data?

Then finally, there is your decoding process.  Which software is converting AIS RF received bits into messages and finally NMEA and optional metadata (e.g. NMEA TAG BLOCKs)?  What library is decoding the NMEA and metadata?  What is being done to check timestamps find probable errors.  If your research is just on the decoding XML, json, or whatever decoded format, how can you verify that their decoding / interpretation of the original AIS is correct?  Perhaps they interpolate position reports (or did for just a bit of time) to make their system look more impressive?  Or their system rounds values or bugs in their processing snap positions to locations.  And so on and so on.

And there is also noise throughout the entire process.  The checksums in NMEA are pathetic.  And many encode and decode systems will just drop messages that don't meet their expectations.

Yes, you can spoof the GPS system, but the above shows that there are plenty of ways to have craziness in AIS data that are not caused by GPS spoofing.

Time to take a look at the article...

https://nrkbeta.no/2017/09/18/gps-freaking-out-maybe-youre-too-close-to-putin/  Skrevet av Henrik Lied 18. september 2017

While their timing with Putin's location is pretty convincing, I'm mostly left with questions.  Were these all Class A devices or where some Class B devices impacted too?  Were their any logs of positions from other devices?  e.g. a cell phone with GPS off, but able to see wifi can give you position estimates.  Were there any AIS satellite message (27?) position reports impacted?  What were the configurations and models of AIS and GPS for these ships?  Where there issues with other received AIS messages?  e.g. if you had any fixed base stations or ATONs reporting position, did they stay fixed and what positioning mode were they in (e.g. GNSS or surveyed?)  What happened with ships at dock or anchor in the region?  What was reported by different GPSes on the same ship?  Usually the AIS has a GPS receiver inside the device that is separate from the chart plotter.  Were any NMEA logs from the ship recorded?  Were their heading or commstate quirks in the AIS messages?


http://maritime-executive.com/editorials/mass-gps-spoofing-attack-in-black-sea

They quote from the master of the ship:
Thank you for your below answer, nevertheless I confirm my GPS equipment is fine.
We run self test few times and all is working good.
I confirm all ships in the area (more than 20 ships) have the same problem.
I personally contacted three of them via VHF, they confirmed the same.
Sometimes, position is correct, sometimes is not.
GPS sometimes looses position or displays inaccurate position (high HDOP).
For few days, GPS gave a position inland (near Gelendyhik aiport) but vessel was actually drifting more than 25 NM from it.
Important: at that time, GPS system considered the position as "Safe within 100m".
Really?  "my GPS equipment is fine"  How is a master supposed to really know that?  What do those self tests do?  What does their system do to declare that safety statement?  I'd love to see the raw log from even a regular fixed GPS for a time window that included the time of issues.  Even better if someone with a 100+ channel survey grade receiver had RINEX logs.  What does the phase tracking info say?

Summary

Really, these articles are just teases.  There is no really useful information in them.  The topic deserves a real research project with data published along with the results.  I encourage researchers out there to dig into these issues.

I also wonder about these hardware and software on the ship and receive stations.  I've never fuzz tested an AIS transceiver (RF input or ship comm channels), GPS, or ECDIS/ECS.  I've poked at fuzzing libais and it definitely needs work.  Also, some of these systems have OSes under the hood that could easily be attack via more traditional malware or vendor supplied issues.  e.g. If you have Transas ECDIS on your bridge that only runs on Windows?  Your machine could be pwned or Transas could have worked with the Russian government to have a way to allow the system to degrade positing when it wants... I variation on the US's ability to turn on Selective Availability (SA).

Note: I am NOT claiming that Transas has done this.  It's just they are a Russian company.  Correct?  I can't find any info to fact check my memory.

I am continually baffled by the maritime communities' believe that they are somehow special when is comes to computer security.  My message to all those working with ships:  Your IT security issues are not unique or special.   Your just worse at dealing with them than most and have your head in the sand even more than Equifax.