Friday, March 24, 2017

Danish AIS almost interesting

I just saw that the Danish have released AIS data publicly.  I was excited until I saw what one of the files looks like.  It's not RAW AIS NMEA data.  That means it's crap for anyone wanting to do use it for rigorous research into AIS.  You don't get to see the station info, the comm state, or any of the interesting channel messages.  You won't get to see each of the ship name messages.  Or the meteorology/hydrology messages.  You also won't be able to do anything if you find they have a processing bug or don't like the way they choose to conflate msgs 1-3 with 5.  Also, what exactly is the license / copyright status for the data? Big bummer.

That's in addition to the already annoying marine link story that doesn't actually link to the data.  MarineLink does an amazing job of annoying me.

And they named the files MMMYYYY so they don't sort nicely on their ftp server (wait, ftp?  what year is this?)  Extra bonus of using a zip so I can't zcat/bzcat/xzcat a partial download to see what I'm getting into.  At least the most recent data is available by day as an uncompressed file.

Thursday, January 26, 2017


Just to keep up with things, here is the latest.  I've definitely failed to blog enough this week.

Power GEODSS Sears Tower Social media Ti computer terrorism Task Force
nuclear Tehrik-i-Taliban Pakistan Gulf Cartel Listeria MARTA NATIA
JAVA Trump industrial intelligence 42

4c a7 9b 81 b9 3a ae 99 b1 40 a6 1f 08 c9 4e 43
41 2f 7a 4b ac 8a 5f 70 f9 60 a1 3e 79 dc da 1f
fb 97 19 72 6f 20 81 15 a5 04 c0 52 45 ae 86 7b
60 e6 71 6e ad 32 70 81 5b 30 8c 24 5f 83 d2 78
5b 32 f0 8f 1b 1a 16 3b f7 02 fc 87 df 02 0d ac
d0 32 c2 b0 d7 f2 08 1f 0d 87 06 9b 68 18 be e0
51 dd f3 38 a5 34 90 cd 64 8f ea f2 57 2b ba f1
ba 1e ff 22 9f 72 f0 3d b7 f2 82 52 f6 be 56 ea
70 24 62 b0 f4 bc b0 c0 89 34 62 2b 9d 12 44 09
f3 2b 5b 3c 04 e1 e8 bc 34 02 90 31 a4 92 89 38
2e 1d 82 c4 de fe 0c ca 81 b5 73 2c f7 70 db 35
96 c1 34 01 76 26 24 6b c6 df 64 29 74 90 ab bd
a1 5d 5a b6 5d 34 58 09 d2 3a 76 d5 93 d8 3c 67
a7 38 fd ef f0 f1 24 49 b7 76 94 9d fa 0e fe 75
02 f4 38 ef a1 36 0c 7e eb ac 7b a6 26 b6 24 11

88 24 2a c2 2c bc eb fb 1f eb 2e 06 99 28 7f 42

Sunday, January 8, 2017

Code reviews

I've sent out thousands of changes for code review and done reviews on many hundreds of changes.  It's kinda weird to read this RedHat post on code reviews.  I can't really imagine what the review process is like without things like the Google C++, Python and Java style guides (I haven't done Go, sorry Francesc).  Things like that and required automatic formatting rules take a lot of the pain out the review process.  But I have had changes that have taken more than a year to get into a code base.  That does get a wee bit tiring sometimes.  Pythonic code review (Red Hat Security Blog)

MIT 046 Algorithm course

I'm going through the MIT 046 algorithm course from 2015...

Sunday, January 1, 2017

Blogging again

I am going to see if I can get back into blogging again. Can I pull off a post a day for January?  What do I even want to write about these days?  A lot of my work is non-public Google infrastructure work.  I can take about some of the open source work and maybe I can talk about some of the public data I am around. I need to start digging into JTS, Geotools, and Java in general. I want to get more into algorithms and data structures.  These last couple of years have been paying down massive technical Dept, which isn't sexy. And buffet overruns and hard to debug code is no fun. Lately I have been playing with Flume and writing and reviewing some fun modern C++. But... Being a parent tends to dominate free time. If Linc could choose, I would probably blog about trains, King Bob, and wheels.  Or the things we see in the Santa Cruz area like this Red shouldered hawk we say today.

And can I figure out how to get the blogger app to work well on Android?  Once I had added a picture, everything went south.  I finally ended up copying the text from the app and taking it over to my laptop to use a real keyboard and the blogger web interface.

Monday, October 10, 2016

Maritime cyber security

Please someone tell me why this is not just totally stupid.

"Cybersecurity incursions and threats to the Marine Transportation System (MTS) and port facilities throughout the country are increasing," said Dr. Hady Salloum, Director of MSC. "This research project will support the missions of the DHS Center of Excellence and the U.S. Coast Guard to address these concerns and vulnerabilities and will identify policies and risk management strategies to bolster the cybersecurity posture of the MTS enterprise."
I don't think there is anything unique about the maritime environment when it comes to computer security.  It's just like other industrial control situations with the twist that you have people living at the "site."  That isn't really even a twist as there are lots of jobs where people live at the site.  The main problems I have seen or heard of on ships are no different than just basic computer security issues.

I take this a bit personally because of an incident back in 2002.  I was on the RVIB Palmer (NSF's ice breaker) as a scientist.  The head of ship board computer pulled me aside one morning with the statement, "If you tell me what you did, I won't press charges."  Needless to say, I had no idea what he was referring to.  After quite the standoff, he finally revealed that most of the ship's network had gone down.  He was accusing me of hacking into ships systems and trashing things.  After even more of a standoff, he said the logs show me breaking in.  I tried to explain all that I had done on the ship, which was to, with permission from the other tech first, log into a machine with the navigation logs on it and write a shared memory reader program to write out a csv file of the ship's track.  He didn't believe me.  He said I was the only person on the ship that knew how to do this kind of malicious thing.  I finally turned my back on him and walked off.  Many hours later, he found me to apologize.  He said a Windows virus got in through someone's email and had run rampant on the system.  At the time, I only had a linux laptop and was using the Solaris workstations on the ship...  he said he now understood that it wasn't me.  I very much appreciated the apology with the accompanying explanation, but not his initial reaction.

So when it comes to comes to computer security on ships, the things I see are:

  • Get the software vendors to provide systems that are better tested and audited.  Which ECDIS has user visible unittests and integration tests?  Which use all the available static analyzers like Coverity and Clang Static Analyzer? 
  • Force all the specifications for software and hardware to be public without paywalls
  • Consider using open source code where anyone can audit the system
  • Switch away from Windows to stripped down OSes with the components needed for the task
  • Teach your users about safely using computers
  • Run system and network scanners
  • Ensure that updates are done through proper techniques with signed data and apps
  • Hardware watchdogs on machines
  • Too many alarms on the bridge for irrelevant stuff
  • Stupid expensive connectors and custom protocols blocking using industry standard tools
Common issues I've heard of:
  • Crew browsing the internet and watching videos from critical computers getting viruses
  • Updates of navigation data opening machines up for being owned
  • Corrupt data going uncheck through the network (e.g. NMEA has terrible integrity checking)
  • Windows machines being flaky and operations guides saying to continuously check that the clock changes to machine sure the machine has not crashed
  • Single points of failure by banning things like hand held GPSes or mobile devices
  • Devices not functioning, that are impossible to diagnose / fix except by the manufacturer
  • Updates taking all the ships bandwidth because every computer wants to talk to home
  • Nobody on the ship knowing how systems work and the installer has left the company
  • TWIC cards... spending billions for no additional security
  • No easy two factor authentication for ships systems
  • Being unable to connect data from system A to system B cause of one or more reasons
    •  No way to run cables without massive cost
    • Incompatible connectors
    • Incompatible data protocols
    • Proprietary data protocols (e.g. NMEA paywalled BS)
  • Inability to do the correct thing because a standard says you can't.  e.g. IMO, IALA, ECDIS, NMEA, USCG, etc. is full of standards designed by people who did not know what they were doing w.r.t. engineering.  Just because you are an experienced mariner, doesn't me you know anything about designing hardware or software.
  • Interference between devices with no way for the crew to test
  • Systems being locked down in the incorrect configuration.  e.g. USCG's no-change rule for Class-B AIS
  • No consequences incorrect systems and positive feed back systems for fixing them.  e.g. why can't it be a part of the standard watch guides to check your own AIS with another ship or the port?  The yearly check of AIS settings guideline is dumb.
  • People hooking the wrong kind of electrical system to the ship
  • No plans to the ship, so people make assumptions about what is where and how it should work.  As-builts don't have xray vision.  Open ship designs would be a huge win
  • Ability of near by devices to access the ship's systems
  • And so on....
Sounds just like cars, IOT devices, home automation, etc.

Saturday, October 1, 2016

Autonomous ocean mapping

Back when I was in grad school at SIO(somewhere in the 2001-2004 time frame), I proposed an autonomous surface vessel to map the southern oceans.  I was resoundly told it was a terrible idea.  I was thinking diesel + solar powered, but now I know that there are a good number of other alternatives.

It's great to see that the idea is now getting notice.  It really is much more cost effective than sending manned vessels.  People are expensive and it takes a lot to keep us alive.

Currently, the most efficient way to map involves the use of multibeam sonar, which sends pulses of sound that bounce off the seafloor and back. Autonomous underwater vehicles can also be used, though they are less efficient. At the meeting, Mayer floated the idea of an unmanned barge, equipped with multibeam sonar, that could roam the seas while continuously mapping, which would cost about one-third as much as a manned vessel.
I wouldn't say "barge", but things like Argo floats could be made larger and have single beam sonars.

Also, it would be great to have larger ships have depth sounders that were capable of deeper water, but that is tough with the shipping industry having such thin margins and massive ships lasting as little as 11 years before they are sent to the breaker.

I was sure that I had written about this someplace before, but I can't find it.  I need to snap a picture of my early thoughts on the topic and add it here.  Some of which is totally irrelevant now.  If you used a wave glider type system, there is no need for propulsion.