Wednesday, November 30, 2011

RT 25 - R for Statistics

Rob Braswell was our guest lecturer covering R for statistical analysis.

mp3, pdf of screenshots, 25-R-lab1-Intro.pdf, (on bitbucket via hg).

Wednesday, November 23, 2011

RT 24 - Part 4: Parsing binary SBET files with python's struct

Topics include "What is GIS," using glob.glob to expand file names with "*", using sys.argv directly to list input files and using argparse to properly handle command line arguments.

html, pdf, mp3 and org (in BitBucket hg).

Saturday, November 19, 2011

RT 23 - Part 3: Parsing binary SBET files with python's struct

In this class, we do our first mercurial pull of changes. We then add helper functions to our module to give us the number of datagrams in an sbet file, tell us at what offset any particular datagram is located and add a generator function allowing cleaner for loops over sbet files.

23-python-binary-files-part-3.html, mp3 and pdf

Remember that at this point, you should be getting the org mode formatted class notes via mercurial (hg). To get set up:
mkdir ~/projects
cd ~/projects
sudo apt-get install mercurial # Install hg on ubuntu & debian linux
hg clone

And every time you start working on the class, do a pull and update to get the latest versions.
cd ~/projects/researchtools
hg pull # Bring the changes down to the local "repo"
hg update # Change the working files to have the latest changes

Wednesday, November 16, 2011

RT 22 - Part 2: Parsing binary SBET files with python's struct

Rob Braswell has tentatively agreed to give us a lecture on Data Analysis with R on Nov. 29th. Rob taught EOS 864 while he was a full-time Research Professor at UNH. I haven't done any work with R myself, but I've seen some really great work done with it. I'm super excited to sit in on his class.

Notes for the class should now be retrieved via mercurial/hg from The rest of the material is in the usual locations in the class directory: mp3, pdf, html

Friday, November 11, 2011

Sunday, November 6, 2011

RT 20 - BAGs 3

Using python .format template language, writing KML, using GSHHS global shoreline to give context to the bag bounding box. Given on 2011-Nov-08.

org, html, mp3 and pdf

Thursday, November 3, 2011

RT 19 - BAGs 2, XML Metadata

KEYWORDS: BAG HDF HDF5 XML lxml etree hydrographic survey raster metadata shapefile

html, org, mp3 and pdf

Friday, October 21, 2011

Friday, October 14, 2011

Research Tools Lecture 13 - if, while, functions, classes

13-python-if-while-def-class.mp3 and org, html, and pdf

I go over boolean expressions (1==1 gives True and 1==2 gives False),
while loops, functions, classes, and start into collecting data with
socat from a weather station on the roof of our building.

Wednesday, October 5, 2011

Research Tools Lecture 11 - ipython and python data types

org, html, mp3, pdf

Research Tools Lecture 10 - QGIS, Bash script, Healy movie, ipython, matplotlib

org, html, mp3, pdf

Research Tools Lecture 9 - Babel, bash, Google Earth

org, html, mp3, pdf

This is the first lecture where I realized that I should be capturing screen shots from my virtual machine while I teach. I put those into a keynote/powerpoint/pdf.

Research Tools Lecture 8 - More emacs and script files

org, html, mp3

Research Tools Lecture 7 - Emacs and org-mode

org, html, mp3

Research Tools Lecture 6 - KeePassX and Dropbox

org, html, mp3

Things to change:

  • The filenames need to change from keypassx to keepassx!

Research Tools Lecture 5 - Filetypes, Intro to Emacs, Beginning Scripts

5-filetypes-emacs: org, html, mp3

Research Tools Lecture 4 - VMWare Ubuntu Image

4-ubuntu-virtual-machine: org, html, mp3

Research Tools Lecture 3 - Wiki editing, Weather Demo, Basic Cmd Line

3-basic-command-line: org, html, mp3

Research Tools Lecture 2 - IRC, MediaWiki, Basic Shell

2-irc-wiki-basic-shell: org, html, mp3

Research Tools Lecture 1 - Introduction and 1-introduction.mp3

This is a place where people can discuss lecture 1 of research tools. I'm interested in questions, clarifications, corrections, suggestions, etc for all the lectures. I'll do a post per lecture and video. However, the videos are on youtube, so commenting there is best.

The course material is all in emacs org-mode and in later lectures, I will use org-babel to execute code from inside the document to follow the concept of reproducible-research and "tangling" documentation and code. The Research Tools class website has HTML versions of the notes too if you are not interested in the emacs aspect of the course.

I am creating the lectures on the fly (despite trying to get ahead over the last year), so any feedback I get will help me improve the lectures for the rest of the semester.

Please note that I edit the audio files down from the full hour and 20 minutes to about 50-60 minutes. I clean up some of my "um"'s and "ahh"s along with white space, repeats and other non-useful bits. I walk around the class and there I often explain the same or similar concept to multiple people. I try to reduce the repetition. It's helpful in class, but with the audio, you can just rewind it.

The youtube videos are unedited, so they will show more of my typos.


Thursday, July 28, 2011

Turned off adds

I was playing with adds just to see what it was like. I reached $13 since I started adds and have not cached that out. I'm visiting Google, it does not seem right to have adds on my blog (unless there is a way to point the money at a charity of my choosing), and they were a bit overwhelming the way I had them setup.

Geotagging in Blogger

Draft Blogger now has geotagging!  Where am I?  I'm a Visiting Faculty at Google at the moment!

Tuesday, June 14, 2011


command line posting

Wednesday, June 8, 2011

Examples of great metadata?

Today, I am sitting in a metadata workshop at UNH run by Northeast Coastal and Ocean Data Partnership (formerly: Gulf of Maine Ocean Data Partnership) NeCODP... it's also their annual meeting. Back in January, I attended a Metavist metadata workshop put on at the Alaska Science Symposium by NBII's metadata folks.

I have been struggling for a long time with metadata. I have felt strongly that something is fundamentally wrong. I'm starting to feel that this is that we are missing examples that are held up to the community as best practices and examples of how this metadata is used. As Google has popularized, we need to dog food the metadata. And it has to be understandable by the 1st year graduate student, not just metadata experts!

So, what are example of best practices in ISO 19115 metadata (I consider FGDC metadata as dead and pointless for a global community). And it really sucks how closed the ISO standards are. So totally frustrating. Am I going to try to teach UML to new grad students? We need to be able to kickoff with metadata and get going in 15-30 minutes and UML is counter to this understanding in short time frames.

The keynote speaker is Ted Habermann who gave a fantastic talk. However, strongly disagree with Ted on a couple points. First is with the pay for standards model. We are not making light bulbs and the standards bodies are not paying us for our time writing these standards. Hiding software and data standards only degrades the ability of individuals to be empowered. Large companies like ESRI could care less one way or another about the costs, but it is up to us, the data producers and consumers to be empowered. It's getting way way too complicated! Ted's Metadata Standards page doesn't even cover a small fraction of the standards he talked about during his keynote. e.g. what is EML? He also mentioned UDDC - Unidata Data Discovery Conventions, NetCDF Markup Language (NcML), etc.

There has got to be something simpler, because with complexity comes increased error rates and all sorts of training problems. Can we use simpler strategies? e.g. Geodata Discovery and geo micro format

And to be different than the speaker, when it comes to researchers and especially graduate students, the do go to wikipedia to read multiple papes and you can make books out of groups of pages. Wikipedia:Books

Monday, May 16, 2011

Examples of good cruise reports and descriptive reports (DR)?

I am on the hunt for great cruise reports and hydrographic survey Descriptive Reports (DR). I'm asking a few people directly for their opinions, but here I'm asking anyone who is willing to share their take on examples of what makes a great report.

For example, Jim Gardner shared his CRUISE KM1009 May 17, to June 16, 2010.

Tuesday, May 3, 2011

MS photosynth

Just made a panorama with my iPhone!

Tuesday, March 29, 2011

C++ testing 2011

I started really digging in to C++ testing with a post listing lots of options. If you have comments on any of these systems and especially if you can link to people reviewing their experiences with the testing systems (preferably in the last couple years), it would be a huge help.

My new post: unittesting c++ in 2011 - part 1. My old post last year: Which c++ logging and unit test framework(s)?

Thursday, January 20, 2011

Wednesday, January 19, 2011

Deepwater Horizon Lessons learned

If you are in Anchorage, AK this evening, come by the Captain Cook Hotel and join us for Lessons Learned from the Gulf of Mexico oil spill from 5:45-8:00PM at the Alaska Marine Science Symposium. Here are some lessons learned in my oppinion to try to help get the discussion going:

  • Ships of all sizes should have AIS and there must be a way to track which ships are a part of a response

  • Have portable AIS systems ready to put on ships (suitcase version)

  • Know the bathymetry and be willing to collect your own if necessary. Have charts, the Coast Pilot and local names

  • Have a manual that you get to all of the team. People coming to the area don’t know the local names, physical layout, and available data / resources

  • Start using mobile devices to collect data – position and information. Avoid handwritten and voice communication to prevent confusion

  • Train with actual ships being virtually moved around

  • Get necessary data integrated into AOOS, ERMA and other systems before you are in a crisis. Unlike Deepwater Horizon, initial response time is usually critical

  • Train people to work with the data types before a crisis

  • PDFs and paper are not good primary delivery devices

  • Communication across all the teams makes or breaks the response

  • We should have AIS from a couple response vessels forwarded through satellite to supplement normal stations

Some are being worked on now, some are not. What would you add?

Sunday, January 2, 2011

How to learn emacs lisp?

During 2011, I would really like to start writing org-spatial for
emacs. I spend most of my life in emacs and it is crazy that org-mode
and emacs do not know about location and projections. If that is going
to change, I'm going to have to step up and get it started. However, I
suck at reading lisp. What is a great source for getting going? ShowMeDo
only has one video tagged with emacs.

I just found rpdillon on youtube. I just wish he had more videos.

How would you recommend to a total LISP beginner that they get into org-mode programming?