• OS X Mountain Lion
• MacPorts
• PostgreSQL 9.1

Install PostGIS

Golly, I recently wrote a tutorial how to install PostGIS - what a coincidence.

Install GDAL

GDAL is a “translator library for raster geospatial data formats”. The important part is that it comes with a commandline tool to convert from one format to another, named ogr2ogr. We want that:

$ sudo port install gdal +postgresql91

Check that it works:

$ ogr2ogr --long-usage | grep PostgreSQL
-f "PostgreSQL"

Import data to PostgreSQL

Importing a single file using ogr2ogr is simple (and doesn’t require the +postgresql variant above):

$ ogr2ogr -append -f PGDump /vsistdout/ gpx.jsp\?relation\=1076755 | psql DATABASE_NAME

Using find we can easily do it for multiple GPX files:

$ find . -name \*.gpx -exec ogr2ogr -append -f PostgreSQL "PG:dbname=DATABASE_NAME" {} \;

See if it worked

The following should return the number of files you’ve imported (I think - at least in my case it fit):

$ psql -d geo -c "SELECT COUNT(*) FROM routes"
 count
-------
   289
(1 row)

Happy geo-analysis!

• OS X Mountain Lion
• PostgreSQL 9 installed via MacPorts

Step 0: Install PostgreSQL

If you don’t have PostgreSQL already installed, it should be a simple matter of getting it via MacPorts:

$ port install postgresql91 postgresql91-server

Also, I am sure there are tons of tutorials out there to help with this, so I won’t cover that here.

Install PostGIS 1.5

Simple enough:

$ port install postgis

Note that there is a PostGIS 2.0 out there, but it requires PostgreSQL 9.2 and I wasn’t ready for upgrading just yet.

Setup a geospatially enabled database

This part had me a bit stumped, and digging up the right file paths for this to work took a while:

$ createdb geo
$ createlang plpgsql geo
$ psql -d geo -f /opt/local/share/postgresql91/contrib/postgis-1.5/postgis.sql
$ psql -d geo -f /opt/local/share/postgresql91/contrib/postgis-1.5/spatial_ref_sys.sql

If you figure you need to routinely setup new databases like this, it might be an idea to create a database template to use in the future.

Test that it worked

The following should output a ton (400+) of functions:

$ psql geo -c "\df" | grep st_

If it did, you’re good to go. If it didn’t, good luck.

Here’s an idea; You enter a Twitter hashtag (or any search, I guess), and have the resulting tweets replayed in real time as they would have appeared had you been following the hashtag when the event occurred. That way you get the full Twitter-augmented experience when watching anything after the fact.

A feature to speed up and slow down is probably a good idea as syncing is going to be a challenge, but overall it doesn’t have to be synced with millisecond precision.

It would also be interesting if you could add your own tweets to the timeshifted stream after the fact, but I am not entirely sure how that’d work. Perhaps adding a machine tag indicating a timeshift factor or timestamp could handle it. It would be pretty confusing for your followers seeing your out of context tweets appearing, though.

colors.stub(:favorite).with("Person A").and_return("Mauve")

It is also easy to have different stubs for different arguments:

colors.stub(:favorite).with("Person A").and_return("Mauve")
colors.stub(:favorite).with("Person B").and_return("Olive")

However, the other day I needed to stub out the same method many times for a fair amount of different arguments and repeating the foo.stub(...).with(...).and_return() part gets tedious real fast.

Thankfully, RSpec stubs have a way around this.

We can use this to setup a look up table inside the stubbed method to give is pretty granular control over what it returns:

colors.stub(:favorite) do |argument|
  values = {"Person A" => "Mauve", "Person B => "Olive"}
  return values[argument]
end

This way, whenever your tests call colors.favorite the argument they pass along is used to look up the returned result in the Hash inside the stub.

This has proven quite useful when your FavoriteColorService talks to a DNA sequencer live and we don’t want each test run to wait 4 weeks for each result.

With help from the Ruby user groups in Copenhagen and Aarhus I’ve compiled this list of danish sites/apps built on Rails and launched (more or less) in 2012.

See the list of Ruby and Rails apps launched in 2010.

1. 5 ways to run commands from Ruby
2. Pepper - A Redmine theme
3. no such file to load — mkmf
4. Redmine plugin: Github Hook
5. Hello Ruby on Rails World
6. Using Google Page Speed to optimize websites
7. CSS, links, gradients, rounded corners, and IE
8. Hello Rails 3 world
9. Rails 3: Responding with Excel
10. Ruby: How to check if a String is numeric

As always, the list is dominated by posts written before 2012. Not surprisingly, but it’s always funny seeing this long tail of old content.

Most popular posts from 2012

Looking at only stuff written in 2012, the list looks like:

1. CSS, links, gradients, rounded corners, and IE
2. Rails 3, ActionMailer, and Google Apps for Domains
3. Ruby SSL certificate verification woes
4. Flash is not the 99%
5. All lights are green
6. Designers solve problems - what do developers do?
7. How we took our tests from 30 to 3 minutes
8. 48 hours of commits visualized
9. Fact-checking the 40%
10. SSL certificate woes with Ruby 1.9 and OS X

Referring sites

Hey, it looks that social media stuff is finally taking off: That Twitter thing has made its way into the top 5 referring sites! Just barely.

The Google still reigns supreme as my number one traffic generator, which I suppose, fits perfectly fine with the long tail of old posts mentioned above. Google is way better at digging up old stuff when it’s needed than any social stream service. The latter tends to excel at the here and now.

Stack Overflow has also appeared on the list. Understandably, when you consider many of my posts - and indeed of the popular posts - are very concrete issues one might encounter when programming.

Bing is holding steady at number 10 referring site. That’s below Korben - good for you, Bing…

Browsers

Ah, yes, browser stats from a geek website, always so very applicable to every other site in existence… Nevertheless, here’s the top 5 browsers on mentalized.net:

1. Chrome (53%)
2. Firefox (29%)
3. Safari (9%)
4. Internet Explorer (5%)
5. Opera (1%)

Poor Opera. And while it pains me to say this, having been an avid Firefox user for years, I am kind of surprised seeing Firefox that high on the list.

Onwards and upwards

I know my 2012 was great, here’s hoping 2013 will be just as terrific for all of us.

While we are currently writing most our new tests in the fast test style, rewriting 20.000 lines of slow tests isn’t something that’ll be done in the near future.

So we needed to do something differently that would work here and now.

TL;DR

• An irresponsible database and more hardware

For starters, we rented a dedicated server. Yes, not a cloud thingy, but geniune physical hardware that we could call our own. Surely, 2 quadcore CPUs, 7200 RPM disks, and 16GB of RAM is going to make our test suite scream, right?

Wrong.

Running the test suite on the new server proved only a few minutes quicker than on the tired, old VPS of yore. Disappointing.

Parallelize

The mediocre result seemed to suggest that we were CPU bound. Luckily, with 8 cores in the server, that would be easy to solve.

Enter the parallel gem and with almost no other changes our specs would run on all cores on the machine shaving almost 50% off the run time.

Better, but still disappointing. I expected going from 1 to 8 cores to prove more than a 100% performance increase. Next up, tuning the database.

PostgreSQL, I’m looking at you

PostgreSQL is a great database. Fast, reliable, always making sure your data is correct and consistent. All great features of a database running in production.

However, for testing, we don’t care about our data - it’s throwaway junk generated from fixtures and factories anyways.

Luckily PostgreSQL allows us to configure some of it’s durability away. We changed our postgresql.conf to include the following settings:

checkpoint_segments = '9'
checkpoint_timeout = '30min'
fsync = 'off'
full_page_writes = 'off'
synchronous_commit = 'off'

Basically this disables/delays a bunch of safeguards that PostgreSQL has in place to ensure your data doesn’t get corrupted in case of system crashes.

This was like putting Magic Boots of Blinding Speed on our test suite: 2 minutes and 49 seconds! Good riddance.

What more could be done?

A couple of things I haven’t gotten around to, but that could be interesting to see the results of:

• Running the server on a SSD. I seem to recall this having a huge effect on running the test suite locally, I’d expect it to at least do something on the server.
• Running PostgreSQL from a RAM disk. While my initial tests of this seemed to make no difference whatsoever, I am inclined to chalk that up to me doing something wrong.

Next up, making Jenkins build pull requests and notify us on Github.

Sure, I could import the data into a database and run SQL queries on it, but that gets tedious after a few runs - unless, of course, if you automate it. Which I did, and I have decided to share the tool as CSV Query.

CSV Query is a command line tool that allows you to run SQL queries on data stored in CSV files, for example:

$ csvq --select "count(*)" --where "name='Jakob'" sample.csv
count(*)
--------
       1

Installation

CSV Query is distributed as a Ruby gem, so installation is a matter of

$ gem install csv_query

… assuming you have Ruby set up.

The alternatives

A few days ago, I tweeted:

1. Find problem
2. Fail to find existing project that solves it
3. Write it yourself
4. Find 2 other projects that does the same thing

That was roughly my experience with CSV Query. A few other projects aim to do roughly the same thing, and I didn’t find them until after I had already written my own tool. Sigh.

• csvq.py - Python variant that does almost the same thing, but uses Python for querying
• CSVql - A query language for CSV files

CSV Query is slightly different in its approach, basically passing all querying on to SQLite. Here’s hoping it’ll prove useful for a few people other than myself.

Let’s assume I built something and the total cost of development is a nice round number like $10.000.

Let’s assume I expect to sell a certain number of copies, say 1.000.

Let’s assume I like to turn a profit (I do) and not give away my work (I don’t). How much should I charge?

$1? Don’t be ridiculous.

$10? Assuming every copy is sold that’d put me at break even. But at $0 net I might as well not have built anything. And if I don’t sell 1.000 copies, I’d have lost money.

$20? Now we’re talking. I’d be looking at recouping my costs and making a decent profit - I might even be able to pay for food and housing for a few months.

$100? If 200 customers find my something worth that much, why not?

Now, let’s assume the number of copies with certainty cannot exceed 1.000 for whatever reason.

How much would you charge?