Ben Nolan

Big MySql Imports

2011-11-07T00:00:00-08:00

Today I’ve been fighting a 10+ gigabyte mysql import. I’ve come across and solved a few problems that might be handy to someone in the future.

max_allowed_packet

In your /etc/my.cnf, it specifies the maximum packet size. I had the problem where some of the insert statements exceeded 1 megabyte in size, so increasing this limit to 64M solved my first problem.

Resuming from mid-way through an import

Next up, I wanted to cat the sql import, but skip all the tables up to table xyz. This I accomplished with a little bit of sed magic:

cat dump.sql | sed -n -e '/`some_table_name`/,$p' | mysql --init-command="SET foreign_key_checks = 0;" database_name

Note the set foreign_key_checks parameter to mysql, this is normally set at the top of your dump file, because we’re truncating the front off the dump file, you need to disable foreign key checks explicitly.

Using pipe viewer

PV is a tool to graph your progress. Instead of running cat and hoping for the best, you get graphical feedback on how long your database restore process will take.

CSV Parser in CoffeeScript

2011-10-24T00:00:00-07:00

I recently needed to be able to parse CSV export from excel, in Javascript. The only way I could find to parse it was a stream-oriented parser, so I came up with this that seems to cope with the majority of strangeness, including multiline cells, quotation marks, and escape and non-escaped cells.

It’s implemented in coffeescript:

class CSV
  constructor: (data) ->
    @raw = data

    @_parse(@raw)

  _parse: ->
    @rows = []
    row = []
    cell = ""

    offset = 0

    mode = CSV.RAW

    while offset < @raw.length
      ch = @raw.charAt(offset)
      adjacent = @raw.charAt(offset+1)

      if mode == CSV.RAW
        if ch == ","
          row.push cell
          cell = ""
        else if ch == "\r"
          @rows.push row
          row = []
          cell = ""
        else if ch == "\""
          mode = CSV.ESC
        else
          cell += ch

      else if mode == CSV.ESC
        if ch == "\""
          if adjacent == "\""
            cell += ch
            offset += 1
          else
            mode = CSV.RAW
        else
          cell += ch
      
      else
        throw "Invalid csv parser mode"
    
      offset += 1

    @rows
  
CSV.RAW = 1
CSV.ESC = 2

@CSV = CSV

This is a good example of code which will block the browser in a big-time way. For example, if you tried to parse a one megabyte CSV file, you’ll probably crash webkit mobile devices, and get script timeout errors in slower javascript implementations. Whoever, because all the looping logic is in a while, it’d be straightforward to make the parser run inside a setInterval loop, and let the browser remain responsive.

Overscore.js

2011-10-07T00:00:00-07:00

I’ve extracted the non-blocking .map and .select implementation that I use in my mobile apps to keep them responsive while processing large arrays. I called it overscore.js and it’s available on github.

Javascript Mobile Apps - Tooling

2011-09-25T00:00:00-07:00

As another part of my series on Javascript mobile apps, I’m going to discuss tooling.

I’ve listed my toolkit before, but here it is again:

jQuery
CoffeeScript
Backbone.js
Capt

jQuery

I use jQuery for the same reason everyone else does. It’s well implemented, fast and fun to use. The well implemented part becomes important when you’re dealing with fiddly implementation details of $.ajax calls. Being able to set headers, read headers and hook the progress events of the ajax object is important to a good mobile app. I use etags when checking for updates in my apps, specifying headers in a convenient hash shorthand is the kind of small detail that jQuery covers so well.

$.ajax {
  url : "http://example.com/endpoint/"
  dataType : 'json'
  timeout : 25000
  headers : { 'If-None-Match' : localStorage.getItem('etag') }
}

CoffeeScript

There are pros and cons of Javascript. The biggest downside I see is that if you’re working on projects that aren’t 100% coffeescript, you’re going to end up with ugliness. You either want to be able to convert all your code over to coffeescript (with the exception of 3rd party libraries), or you want to leave it all as javascript. Doing a combination is going to only cause heartbreak.

If you are sold on CoffeeScript though, it’s a fantastic tool. Using the fat arrow => to ensure class methods are called with the right this scope is an awesome timesaver. The biggest problem with CoffeeScript is probably debugging. This was easier in older versions of the coffee compiler, which had a 1:1 line to line conversion ratio, so model.js:23 was the same as model.coffee:23. Now, unhappily, your line numbers won’t match up, but as long as you can view the javascript output and find the error, you can easily work back and find out which line of coffeescript blew up on you.

Backbone.js

If you want a reason why to use Backbone, just view the apps using Backbone list on the backbone site. It’s a fantastic small piece of kit. It adds events, collections and models in a sensible way that will be familiar to any rails developer, but it does it without trying to pervert Javascript.

Basically, if you’re recieving json data from an endpoint and displaying it using Javascript, you should be using backbone or something similar.

I personally, used to make the mistake of trying to be ultra-minimal in my Javascript, avoiding doing heavy computation or having a lot of code over in the client. I’ve changed my way over the past year or two. If you’re struggling to do something without the correct tools, you’re wasting time, and you’ll probably end up with a half-assed solution. Refactor, break your code out into more classes, and increase your lines of code in Javascript.

Capt

This is my own tool, that I can’t really recommend to anyone else at the moment, since it’s a little bit half-assed and half-finished. But the important thing to do is to have a build tool that you can rely on. You want to be able to use .eco, .less, .coffee and .jst files easily, without having to compile them by hand, or having some makefile you run over and over. Find a build tool that watches your files, recompiles on the fly and includes each file seperately, so that when you get error messages, it’s not just error in bundled.js:31423423, which is going to be pleasantly impossible to debug.

I’d be keen to hear what other developers use for their build tools.

Javascript Mobile Apps

2011-09-24T00:00:00-07:00

I’ve worked on three javascript mobile apps now, one used internally by Lonely Planet, one on the appstore, and another an unreleased site I built myself. I’ve been wanting to write up my notes on development for a while, so since it’s a sunny day and the house is tidy, I’ll put up some notes. First up…

Reasoning

First up, why even use Javascript + XHTML to build your mobile apps? The obvious answer is to get cross-platform support out the box, a well written app will deploy on phonegap to android and iOS with no extra work. Another answer, which is easily dismissed, is the benefits of having a single tool that you can use across more than just your mobile app.

Any business logic you encode in Javascript you can re-use for your desktop client and can be tested on the command line using node.js and jsdom. Where possible, one language is better than two in my book.

Finally, by using javascript, you are free from the tyranny of app-stores. I’ve had apps initially rejected by the appstore, and although they were easily resubmitted and are now selling well, if you’re doing anything “fringe”, that may run afoul of the appstore, having the option to deploy your app over the web as an html5 app is a nice fallback.

Using the new static streetview api

2011-09-19T00:00:00-07:00

So, Google just released a static street view api, which is awesome, since showing a streetview of a place will usually jog someones memory much quicker than a top down view.

The only problem with the API (and it may be fixed in a future revision), is that you need to specify the heading that you want the returned image to be facing. This is a problem, since usually you only have the lat long of the place that you are visualizing, and not the bearing from the nearest road that a google streetview car drove along.

However - in my case, I have a complete set of OpenStreetMap data in my places database, so I could:

Look for the nearest street
Calculate the nearest point on the street
Get the azimuth angle between the nearest point and the destination place
Convert radians to degrees

And get the correct street view images. See this search for curry on Geonear to see what I mean.

Here’s the code I used (rails code using postgis):

<%= image_tag ".../streetview
  ?location=#{place.latitude},#{place.longitude}
  &size=280x200
  &pitch=15
  &fov=65
  &heading=#{place.street_view_heading}
  &sensor=false" %>

And in place.rb:

def street_view_heading
  street = nearest_street
  
  point = Place.find_by_sql(['select st_closestpoint(?, ?) as geometry', street.geometry, self.geometry])
    .first
    .geometry
  
  (Street.find_by_sql(['select st_azimuth(?, ?) as azimuth', point, self.geometry])
    .first
    .azimuth
    .to_f * 180 / 3.14159).floor
end

And we’re away. Very nice, thanks Google. :)

Lerping LatLngs

2011-08-24T00:00:00-07:00

I needed some code for lerping google maps latlongs (for providing a custom streetview navigator). Here’s the function in coffeescript:

google.maps.LatLng.prototype.lerp = (b, i) ->
  lat = (b.lat() - @lat()) * i + @lat()
  lng = (b.lng() - @lng()) * i + @lng()

Etags, Phonegap and Rails

2011-08-23T00:00:00-07:00

In the rankers app that I’ve recently finished for a client, I cached the users data offline, but needed a way to quickly and easy to see if the user had the freshed version of the data.

Downloading database updates

To speed up loading time, and enable offline access, I download the list of places (about 350kb) by an ajax call, and then cache the json and the etag of the latest revision in localStorage. Then when the app launches next time, I send a request for the newest version of the places list, and attach the etag to the request, so that rails can send a not-modified response, and save the users some data.

Here’s the rails pseudocode code I used:

@places = Proc.new do 
  Place.find(:all)
end

response.etag = @etag = [
  Place.find(:first, :order => 'updated_at desc').updated_at.to_i,
  Place.count
]

if request.fresh?(response)
   head :not_modified
else
  render :action => 'index', :mime_type => 'application/json'
end

And in the view:

<% cache(@etag) do %><%= @places.call.to_json %><% end %>

This cached the json (since json generation can be pretty slow in ruby), and sends the appropriate header. To handle this on the client side, I used something like this, to cache the places data, and the etag.

$.ajax {
  url : "http://example.com/some/endpoint"
  dataType : 'json'
  headers : { 'If-None-Match' : localStorage.getItem('etag') }
  success : (data, textStatus, xhr) =>
    if textStatus == "success"
      localStorage.setItem('etag', xhr.getResponseHeader('Etag'))
      localStorage.setItem('placesData', JSON.stringify(data))
    else if textStatus == "notmodified"
      data = JSON.parse(localStorage.getItem('placesData'))
    else
      throw "your toys"

    Places.reset(data)
}

I was going to implement my own version of etags and if-none-match, since I thought I might get heisenbugs with different implementations of xmlhttprequest on different mobile webkits, but in the end, I decided not to reinvent the wheel and use what already exists. So far this has worked well.

Detecting phonegap

2011-08-22T00:00:00-07:00

With my capt apps, I do most of the development and testing using the desktop safari browser, and then move onto testing on actual devices later on, once I’ve got the app mostly working. I use this fragment of code to detect if I’m running inside phonegap, or if it’s a desktop browser, and then instantiate my app appropriately.

function onDeviceReady(){
  window.app = new Application;
  app.start();
}

if(window.PhoneGap){
    document.addEventListener("deviceready",onDeviceReady,false);
}else{
  $(document).ready(function(){
    onDeviceReady();
  });
}

The window.app style of encapsulating my apps has worked well for me in apps I’ve built using capt and backbone, I still haven’t got to the point of encapsulating everything (all classes are copied into the window scope for example).

Duplication detection with postgis

2011-08-20T00:00:00-07:00

I’ve written a particularly gnarly query here that I thought I would share. It’s for iterating over a collection of points that have names, and identifying points that likely candidates for merging.

select
  name, floor(length(ST_ExteriorRing(st_box2d(st_extent(geometry)))) * 1000000) as length, count(id)
from
  places
group by
  name
order by 
  count desc;

Basically, it selects places with similair names, and returns the length of the bounding box surrounding those places. You can use this as a basic test to see if those places should be merged. There a tonne of situations in which this won’t work, but to get you out the gate, it might be helpful.

Google doesn't get the web

2011-08-19T00:00:00-07:00

It must drive the google search team crazy, when they look at the sort of html the rest of the google organisation spits out. Take for example - this google places page. There is no html, no metadata, no microformats, no schema.org support. It’s basically impossible to index. How can they run the worlds biggest web crawler, but generate such obtuse and impossible to index code themselves? It’s astounding I say!

Phonegap

2011-08-15T00:00:00-07:00

I’ve got two apps on the go using Phonegap, and this is just a quick post to say that I can highly recommend the mobile web as a development platform. My list of tools / libraries that I use:

Capt for building the project and compiling / watching intermediate languages
LESS.js for css
Coffeescript
Backbone.js
Underscore.js
My responsive map/reduce underscore.js helpers (yet to be released)
Leaflet for touch-based maps that are GPU accelerated
JSON.js from douglas crockford
jQuery, although I may move to using zepto later on
Phonegap for deploying to Android and iPhone
Makefile for packaging and bundling
Jasmine for speccing
Latlon.js for distance and bearing functions
iScroll.js for fixed position headers and footers
Cross-origin ajax requests to allow local development while accessing remote servers
ECO templates by Sam Stephenson for in-app rendering
JST templates from underscore.js for static html

It’s a pretty nice toolkit. I need to do some more work on capt and put up a sample app so that other people can use my toolkit, but yeah, coffeescript, backbone and less.js are a great way to build mobile apps.

Payroll

2011-07-28T00:00:00-07:00

When you’re using xero and minutedock, you get used to having nice quality tools. It’s like having chisels that were shaped by master craftsmen. And then there’s your payroll package…

Payroll?

Payroll is the software that takes money from my company account, pays the tax man, deducts my taxes as an employee, then pays me in my personal account.

iPayroll

I’ve always used iPayroll. Giles Crisp is a great contact there (tell him Ben sent you), and provides great service. With iPayroll, you authorize them to move the money around and they get access to your company bank accounts. You will be paid every Tuesday without having to lift a finger. Plus it does all sort of HR stuff as well, so if you have employees, you can know know how many days off they should have.

Anyway - Ipayroll is great, but it’s kinda funky looking, and trying to sign up just recently has been a real PITA. Bugs a plenty with the sign-up process.

So I went to the xero payroll providers page.

Smart Payroll

Looks awesome. Until you actually look at the screenshots of the real app, not the pretty marketing page. It looks like it was designed in 1901. It makes iPayroll look a perfectly executed piece of modern art. Pass.

Flexitime

Looks awesome. The app is very xero / minutedock-ish, so I rang them up on their 0800 number and asked:

“Can I pay you money to take money from my account and move the money around?”

“Err - no sorry”

The Flexitime guy told me what I’d have to do to use flexitime to satisfy my obligations as an employer. Basically, set up an automatic payment, modify the AP as necessary, update flexitime, and remember to log into IRD once a month and upload a report that I download from the flexitime site. That’s a lot of remembering.

If Flexitime had put a ‘we manage everything’ option on their website with a nice CSS3 button for me to press and a well form that let me set up my organisation, I probably would have chosen them as my payroll provider. Oh well, back to iPayroll.

Map Reduce in Javascript

2011-07-27T00:00:00-07:00

Here’s the non-blocking map reduce class that I was talking about. It uses at most 75% of the CPU, so that the browser remains responsive while your job is processed. You can destroy the job at any stage.

class MapReduceJob
  constructor: (inputsOriginal, mapFunc, reduceFunc) ->
    utilisation = 75
    inputs = inputsOriginal.slice()

    # todo - replace with a better clone function
    inputs = for input in inputsOriginal
      input

    outputs = []

    @interval = setInterval( =>
      tzero = (new Date).getTime()
  
      # Process at least one
      if inputs.length > 0
        outputs.push mapFunc(inputs.pop())

      # Process more
      while ((new Date).getTime() - tzero < utilisation) and (inputs.length > 0)
        outputs.push mapFunc(inputs.pop())
  
      if inputs.length == 0
        clearInterval(@interval)
        reduceFunc(outputs)
    
    , 100)

  destroy: ->
    clearInterval(@interval)
    delete @interval

You use it like this:

new MapReduceJob(
  ['a', 'b', 'c', 'd'],
  function(input){
    // do something computationally expensive here, like uppercasing the input
    return input.toUpperCase();
  },
  function(outputs){
    console.log(outputs);
  }
);

This is excellent for things like clustering markers on a map, or doing a backbone filtering on large collections. You need to structure your code to be asyncronous.

I’ve got a few things I might do with this code:

Port underscore.js to this continuations style, so that you can use all the existing functions. I’ve already ported the collect and inject functions.
Add a progress() method to the class.
Add support for webworkers for desktop browsers.
Port backbone.js to use these continuations, so that for example, the sort function doesn’t block when sorting large arrays.
Assuming the mapFunction is constant time, measure the number of inputs to process per interval, then stop checking the date function.

It’s pretty interesting stuff, we’ll see how we go.

CORS, Rails and Weinre

2011-07-26T00:00:00-07:00

Doing mobile development on your Android but can’t get any debugging action going? Add this code (from the inimitable Tom Shelfer) to the rails controller that provides your json views, so that you can serve your app on a different port than you serve your API:

before_filter :cors_preflight_check
after_filter :cors_set_access_control_headers

protected

# For all responses in this controller, return the CORS access control headers.
def cors_set_access_control_headers
  headers['Access-Control-Allow-Origin'] = '*'
  headers['Access-Control-Allow-Methods'] = 'POST, GET, OPTIONS'
  headers['Access-Control-Max-Age'] = "1728000"
end

# If this is a preflight OPTIONS request, then short-circuit the
# request, return only the necessary headers and return an empty
# text/plain.
def cors_preflight_check
  if request.method == :options
    headers['Access-Control-Allow-Origin'] = '*'
    headers['Access-Control-Allow-Methods'] = 'POST, GET, OPTIONS'
    headers['Access-Control-Allow-Headers'] = 'X-Requested-With, X-Prototype-Version'
    headers['Access-Control-Max-Age'] = '1728000'
    render :text => '', :content_type => 'text/plain'
  end
end

Then install the MacOS version of weinre and edit Content/MacOS/Launcher like so:

java \
    -XstartOnFirstThread \
    -classpath $BASEDIR/weinre-ui.jar:$BASEDIR/weinre.jar:$BASEDIR/swt.jar \
    weinre.application.GUIMain --boundHost -all-

Then launch weinre (webkit inspector for mobile devices), and add weinre to your web app and reload the browser.

<script src="http://192.168.1.77:8080/target/target-script-min.js#anonymous"></script>

Sorry this post isn’t very cohesive - but that there is a fantastic stack for debugging standalone apps to access existing rails apps. I’m using it to debug my phonegap version of the rankers app.

Something expensive

2011-07-26T00:00:00-07:00

I’m clustering 1200 points on the iphone in javascript in an app I’m writing at the moment. On my iphone 2g it takes 1500ms in a worst case. There are two fixes I’m going to do:

Quadtree

I’ll probably end up implementing a quadtree, or a grid-based acceleration structure, since currently the complexity is currently n squared in a worst case. I’ll let you know how I get on with this.

Break something expensive

This is a generic pattern you can use in javascript (here implemented in coffeescript) to maintain responsiveness in your jobs. You create an interval that runs every 100ms, and it will do at most 50ms of work in each loop, until your data has been processed, then terminates the job.

inputs = [.., .., .., ..]
outputs = []

interval = null

interval = setInterval( =>
  tzero = (new Date).getTime()

  while ((new Date).getTime() - tzero < 50) and (inputs.length > 0)
    input = inputs.pop()
    
    outputs.push(
      doSomethingExpensiveWith(input)
    )
    
  if inputs.length == 0
    clearInterval(interval)
    
    console.log(outputs)
, 100)

I might pull that code out into a generalised form (since I could use it throughout the app) and also add support for chaining inputs and outputs:

inputs.mapReduce(filterFunction).mapReduce(processFunction).mapReduce(outputFunction);

Something like that would be pretty cool, but the above code is the simplified version that works for me.: )

Ajax download progress

2011-07-21T00:00:00-07:00

I’m working on an iphone app that needs 650kb of json data to get started, so I needed a bit of progress display while the data is being fetched. This works on mobile webkit:

interval = null

$.ajax {
  url : "endpoint.json"

  dataType : 'json'

  xhr : () =>
    xhr = jQuery.ajaxSettings.xhr()
  
    interval = setInterval( =>
      if xhr.readyState > 2
        total = parseInt(xhr.getResponseHeader('Content-length'))
        completed = parseInt(xhr.responseText.length)
        percentage = (100.0 / total * completed).toFixed(2)
      
        console.log "Completed #{percentage}%"
    , 50)
  
    xhr

  complete: ->
    clearInterval(interval)
  
  success : (data) =>
    alert(data)
}

It’s a bit painful because of the way you access the raw xhr object in jQuery 1.5, but works nicely. I then store the data in localStorage (as a bit json blob) and we’re away.

Wellington.js

2011-06-27T00:00:00-07:00

I’ll be presenting at Wellington.js on the 13th of July about Backbone.js and my experiences with Capt, my javascript build tool.

That is all.

Webkit Transform

2011-06-26T00:00:00-07:00

I was quite taken with these isometric minecraft renders, so I decided to dig out some of my old game code and try and build an isometric renderer…

A little 3d world

That is created using webkit transforms, the minecraft textures. The heightmap is some perlin noise that I same via getImageData. Now to add some collision detection so you can walk around the world.

Disconnected Sundays

2011-06-24T00:00:00-07:00

Four weeks on from my original post, I wanted to do a follow up about the “Day of Disconnection”.

The “day of disconnect” is now one of my favourite things. I look forward to it all week. It’s the day you don’t get sucked into the computer, and you have no excuses to go outside and do all the wonderful things in the world. And when you don’t feel like going out, you are forced to go out and find something fun to do.

My first Sunday was a day of drinking, but the following Sundays have been a lot more productive. Most recently, my partner and I drove up to the east cape of New Zealand in a little convertible and walked up to the easternmost lighthouse in the world. Another Sunday was spent exercising and collecting driftwood for a big bonfire out the back of the house, and I spent another Sunday doing some vigorous gardening.

So - I recommend you try disconnected Sunday. It’s a truly wonderful thing, and makes your Monday coffee + internet fix even better.

At this rate I’m going to end up making a nicely designed sundaydisconnect.com site. Hah. :)

Jasmine and Backbone

2011-06-14T00:00:00-07:00

I use Jasmine to spec / test my Backbone apps. It can be a pain setting up a Javascript testing environment (unless you’re using capt in which case you get a testing environment for free - but I digress. I decided to pull out some code from an existing app of mine and document a bit of it.

This isn’t best practise stuff, I’m making this up as I go along, but it is the summary of several years of Javascript development on desktop and mobile.

Start test suite:

describe 'places controller', ->
  describe 'search view', ->

I like to nest my suites so that any errors point to the correct controller and view.

beforeEach ->
  window.app = new Application
  window.app.location = { latitude : -45, longitude : 170}

My current app has a bit of an awful hack in that it uses a global ‘app’ instance that wraps up all the application functionality. It is easy to work with, but I don’t like it for testing, since your tests inevitably have side effects. But by recreating a new ‘app’ instance for each test you help defeat some of those side effects.

it 'should handle the truth', ->
  expect(true).toBeTruthy()
it 'should exist', ->
  expect(SearchView).toBeTruthy()

Some sanity checks to make sure the view we’re testing exists, there are no syntax or dependency problems.

it 'should instantiate', ->
  x = new SearchView { collection : new PlaceCollection }
  expect(x instanceof SearchView).toBeTruthy()
  expect(x instanceof Backbone.View).toBeTruthy()

Now we start to do something useful, by instantiating the SearchView. Notice that the view can be instantiated outside of the app, and without a specified element. Backbone will create it’s own div element for the view.

it 'should have render method', ->
  x = new SearchView { collection : new PlaceCollection }
  x.render()
  expect(x.el).toBeTruthy()
  expect(x.el.find('input').is('input')).toBeTruthy()

Now we can start to do some ‘functional’ testing and make sure that the view renders up an input inside my view. This is where you can start to do some poke and prod testing, and make sure the result is as you expect it. Note that I’m running my test suite inside safari, but which something like jsdom, you could run this test suite entirely inside node.js.

it 'should render nearby', ->
  x = new SearchView { collection : new PlaceCollection }
  x.collection.refresh $fixtures.placesNearby
  x.collection.status = 200
  x.render()
  expect(x.el.html()).toMatch /harvard square/i

I use capt to populate my $fixtures object, so I can edit my fixture data in yaml or json, and then poke it into the view and check it renders correctly. This is the weakest selector I could use. Something like:

expect(x.$('a:first')).toMatch /harvard square/i

Might be better, or using the :content selector.

it 'should render searching', ->
  x = new SearchView { collection : new PlaceCollection }
  x.render()
  x.el.find('input').val("Kirkla").trigger('keyup')
  expect(x.query).toEqual("Kirkla")
  expect(x.el.html()).toMatch /loading.../i

You can also trigger keyUp / keyDown and other events using jQuery event object. Note that in my case, this will trigger an ajax call to the server, which must be mocked out on the client so that the ajax call returns immediately and fires the success callback. I use the built in mocking support (spyOn) that comes with Jasmine for mocking.

Travelling

2011-06-13T00:00:00-07:00

I’m writing this post from the above the wing of a 18 seater twinprop plane, with the sun going down to my left, setting over mount ruapehu, which looks to be getting a good coating of powder for this years ski season.

Last month I decided to put my German trip on indefinite hiatus, and I’m going to be honest, it was a hard decision. But now four weeks later and I’m starting to enjoy this life. I’m living in a lovely beach house with some good guys. Travelling to Wellington city regularly to get a fix of friends, tech and cafe culture. I’ve been working with a great group of hackers based in Soho, London - apart from the early it’s an excellent way to work.

The next few months aren’t planned out, but I know I’ll be in Wellington for some of July, house-sitting my sisters lovely new house in Newtown, and I expect there’ll be a mission down to the Southern Island - assuming it’s still there after this newest round of earthquakes :(. Finally, my accountant reviewed my situation and helped me set a goal for the next two years, so maybe I’ll be about to get a piece of dirt to call my own in a year or two.

Anyway - hope everyone is healthy and happy and not persuing their dreams at the cost of their friendships.

Disconnected Sundays

2011-06-11T00:00:00-07:00

Nick at Rankers turned me onto a new movement. It’s disconnected Sundays. I’ve raised it with Sam Minnee from Silverstripe. The goal is to not do any work, or to use the web on Sundays.

Basically, avoid using your computer or iphone at all costs. It freaked me out the first Sunday - what am I going to do to entertain myself? Obviously you can just read books. Or if you feel the need to work on your project you can draw your UI plans with pen and paper - but whatever you do, as long as you’re away from the computer, it’s guaranteed to be a good day, and you’ll come back extra charged up for Monday.

Anyway - so here’s to being unavailable on Sundays and spending it with friends, or with your mountain bike, or working on the garden, maybe jamming in the back yard with some guitars and a drumkit.

Backbone - get or fetch

2011-06-10T00:00:00-07:00

A common mistake people make in Backbone apps, is writing a view that assumes that data exists in a collection, so that when the user reloads the page, the collection is empty and the view won’t render.

For example, you have a Posts collection and a controller:

class Posts extends Backbone.collection
  # ...
  
class PostsController extends Backbone.Controller
  routes :
    "" : "index"
    "topics/:id" : "show"

  index: ->
    Posts.refresh {
      success: ->
        new PostsView { el : $(body), collection : Posts }
    }

  show: (id) ->
    new PostsShowView { el : $(body), model : Posts.get(id) }

The problem with this, is that when you reload the page and you access the index view, it fetches the data with refresh(), then waits until the posts collection has been fetched and renders the view. (This is a bit of a contrived example, but it demonstrates the behaviour).

After the collection has been populated, you click on a post, and it redirects to #posts/12345, a post id that is in the collection. All good.

The problem is when you hit command-r and reload localhost:3000/#posts/12345. The show view will be called straight away, and the posts collection was never populated.

GetOrFetch

I like to fix this with a little helper called getOrFetch:

class Posts extends Backbone.collection
  url: "/posts"
    
  getOrFetch: (id) ->
    if @get(id)
      post = @get id
    else
      post = new Post { id : id }
      @add post
      post.fetch()

    post

Let’s be honest, this isn’t magic, but it means you can replace your show action with:

show: (id) ->
  new PostsShowView { el : $(body), model : Posts.getOrFetch(id) }

Which means that no matter how you get to the show action, your model will be loaded from the server, or just use the local version.

Status

I’ve taken to using http status codes to represent what the status of a model is. For example a model that has been initialized but not loaded is status 0, if it is being loaded it is status 100, if it is loaded it is 200, if there was an error or the id was not found, 500 or 404, etc. This means that your view can have some code like:

<% if model.status <= 200: %>
  Loading...
<% else: %>
  <%= model.escape 'name' %>
<% end %>

Doing this in the mobile app I’m working on at the moment gives a nice ios-ish interface. To make your model status update:

getOrFetch: (id) ->
  if @get(id)
    post = @get id
  else
    post = new Post { id : id }
    @add post
    post.fetch {
      success : ->
        post.status = 200
        post.trigger 'changed'
      error : (e) ->
        post.status = 500
        post.trigger 'changed'
    }

  post

Sunset view

2011-06-06T00:00:00-07:00

I’ve moved out to the east coast of New Zealand to work onsite with a client. I’ve been gardening, programming, brainstorming, prototyping, surfing, swimming and mountain biking. It’s been a nice time.

View out the window from the office

I’ll probably be in Wellington for the month of July, but apart from that I’m free to move around. If you have any interesting Javascript work on later in the year, I’d be keen to do a 2-3 month stint onsite. Anywhere in the world is fine.

Offline Mapping in HTML5

2011-06-03T00:00:00-07:00

I’ve been doing some research and development around offline mapping for a client of mine. Here’s my notes on the current state of technology.

Tile based offline mapping

The easiest way to do offline mapping is to use something like openlayers and store tiles offline. This is kind of gross at first glance, but in practise it works well. It’s very simple, the technology is mature, and above a certain zoom level, it can actually be smaller to store the raster data than the underlying unprocessed vector data. However, it clearly doesn’t work for high zoom levels.

To map an area the size of New Zealand using offline tile caching to zoom level 15, a naive implementation would take about 25 gigabytes of storage. With a little development you could probably get it down to 1-2gb, and with some serious development (adaptive zoom levels based on feature detection in the underlying vector data) you could maybe get to a ~200mb download, but it’s clearly not a trivial download.

However - this isn’t such a big problem if you’re doing a small area like a city or a town, So - assuming you don’t need ultra-high zoom levels, and you can solve the issues of download size, what technology exists to do offline mapping?

Delivering offline maps

The team at mapbox have created a suite of tools for offline mapping. Maps on a stick is an interesting tool that uses openlayers, firefox and html5 to create an offline mapping tool. It is written in javascript, and requires that you download the map tiles and store them on your local filesystem. The downside to this is that copying over 800,000 5kB tiles takes approximately forever. To solve this issue on their ipad app, Mapbox created the mbtiles format, which is involves putting all the maptiles into a sqlite database that modern filesystems can move around much easier.

Sadly, maps on a stick doesn’t support mbtiles, so I looked into what would be involved in adding mbtiles support to maps on a stick. Basically, it boils down to creating a firefox xpcom component (in javascript) that uses the firefox sqlite interface to expose the tiles to openlayers. It’s doable, but fiddly.

On the plus side, you could then wrap up maps on a stick, your mbtiles components and xulrunner to create an installable offline mapping tool. From my experiments with xulrunner and reading through the xpcom documentation, this is non-trivial.

Chrome store?

You could just shortcut all the xulrunner / xpcom stuff and do your offline apps using the chrome app store, or safari html5 offline support. I prototyped this and made a phonegap-powered ipad app that downloaded tiles from cloudmade and stored them in the html5 web database store. The downside to this is downloading the tiles is sloooww, and although you can do multiple requests at once using multiple subdomains, you still get slower throughput than if all the tiles were globbed together into one sqlite database.

Vector data and C++

The best solution, if you were keen to do some serious development, would be to bust out your compiler and package up mapnik or another renderer to act as an xpcom component for firefox. You could still serve tiles, so you could use openlayers (or tile5 or leaflet), and you would have infinite zoom, nice small vector data (the OSM dataset for New Zealand is about 25mB) and you would win all the technical brownie points.

Down this path also lies writing your UI in QT (ala Google Earth) or WX instead of using Firefox, so you could create a completely custom mapping tool.

Analysis

I haven’t finalized my decision yet, but it seems to me that there are no natural inflection points on the cost/benefit graph for offline mapping. If anything, I might lean towards xulrunner + something like “maps on a stick”, but it’s neither here nor there.

Basically, there are no good offline mapping solutions for netbooks / laptops, so maybe there’s a market niche there.

Hutch - tile-based game engine

2011-05-30T00:00:00-07:00

I’ve opensourced the repository holding my tile-based game engine. I worked pretty solidly on it for about two months (and put a bunch of videos up at reddit.com/r/hutch/), but it’s become a back-burnered project, so I’ve donated it to the html5 gamedev community.

The game was written before i developed capt, so all the code is located in public/scripts/ and hosted by rails. It’s designed to work with the cutegod tiles, and it’s probably a real PITA to get the thing set-up and running. I’d like to extract the coffeescript source and convert it into a capt project, so that it’d be easier for people to get the source up and running (and also start adding specs for everything), but don’t hold your breath on that one.

Orthographic 3d

The projection is orthographic, with y and z sharing the same axis (depth is faked using the z-index). The engine is designed in 3d, so positions are expressed as 3-vectors. Collision detection is done in the player model - if I started work on this again, I’d probably go for an architecture where classes have components, instead of a o-o big hierachy tree.

Multiplayer

There is some websocket multiplayer support in the game. The ruby-based server was basically just to prototype and experiment with realtime multiplayer in the browser. In practise I’d probably use a node-based server so that you didn’t have to duplicate code from coffeescript to ruby (for example the vector class, player class, etc, etc).

Future

I’m still very interested in multiplayer games in the browser. I’d love to build something like minecraft, second life or terrarium, where it’s basically a sandbox for people to build worlds and gadgets for other people to explore - but it’s obviously a massive project, and if I did more work on html5-gaming, it’d probably be as part of a team.

Capt Tutorial

2011-05-27T00:00:00-07:00

I got halfway through a tutorial on capt, my build tool for single page javascript applications using coffeescript, backbone.js, eco and less. The tutorial is available here, but it kind of stops halfway down. Feel free to fork the tutorial on github and submit patches, otherwise I’ll finish it up when I have time.

Wellywood

2011-05-26T00:00:00-07:00

I joined the Greens party in the afternoon, met the leadership in the evening and then was interviewed by the film crew at the backbencher - because I support the Wellywood sign. Video below. It was a fun night.

Breakfast or lunch?

2011-05-25T00:00:00-07:00

I give you a time of day, you give me a name for the appropriate meal. I was trying to work out how to do this with a case / switch statement - then came across this syntax which is kinda nice:

def time_of_day_to_meal_name(t = Time.now)
   {
     0..5 => 'Midnight snack',
     6..10 => 'Breakfast',
     11..14 => 'Lunch',
     15..16 => 'Afternoon tea',
     17..21 => 'Dinner',
     22..24 => 'Late night snack'
   }.find { |k,v| k.include? t.hour}.last
 end

What would you like for lunch?

Offline maps

2011-05-23T00:00:00-07:00

I’ve recently had a project morph from an online html5 mapping site, into an application that needs to run offline on netbooks and macbooks. So I’m investigating taking an openlayers, backbone and coffeescript application and packaging it up with Firefox webrunner (or Prism) as an installable package for Windows and os x.

Doing installable software isn’t new territory to me. At Indigo we built installable packages for Windows, OS X and Linux - so I’m pretty happy to work my way through innosetup or nsis, and the os x packagemaker. For the map tiles, I might try and adapt the mbtiles specification from the guys at mapbox. This will be a problem, as the database will have to be jerry rigged into place where Firefox can see it. I’m avoiding doing any binary firefox extensions for this project, although if things go well, there is the posibility of storing vector data locally and creating a mapnik build that operates as a firefox extension.

Data will be stored locally using backbone and sqlite, with regular online syncing to the official point-of-interest repository. So far we only have the html5 protoype done, so there is a lot of ground to cover - but I’ll try and keep you all updated on how things go.

Why Brisbane is the best startup town

2011-05-18T00:00:00-07:00

This post comes as a surprise to me. When I planned to visit Brisbane for 8 weeks to spend some time with my girlfriends parents, I didn’t expect the town would grow on me so much. My previous opinion of Brissie have been is as a cultural wasteland of chromed-up bars and old holdens.

How wrong I was. I’ve come to really like brisbane, and I think it may be one of the best towns to do a startup in Australia / New Zealand.

Great technology scene

There is a great network of javascript, rails and other technology groups. They meet regularly, have some exceptionally strong developers (tile5, sproutcore and several other good opensource projects are maintained from brisbane). If you’re boot strapping, there is good demand for web developers working on modern projects, so you can make a good packet consulting.

Support for startups

The complex of buildings along southbank are invaluable for people working from home or without a fixed office. The edge is a beautifully designed digital workshop where you can grab a couch and free wifi for a few hours. Next door is the national library which has hundreds of modern and beautifully designed workstations that are free. Both universities have modern study halls where you just walk in and grab a desk. Mobile broadband is much cheaper in Australia than in New Zealand, so you can work from anywhere. The weather is better than Wellington or Melbourne so you can easily work outdoors wherever you can find shade and somewhere comfy. You’re just over an hour from Sydney by plane, so you can head down and pitch the collection of investors there, without having to live in Sydney.

Low cost of living

This is a comparative thing, but compared to Sydney or Melbourne (the only other two cities that could pay comparably to Brisbane) the cost of living here is much lower. A room in a cool part of town (within easy biking distance of the city) will cost you about $150 / week, which is less than half of what you’d pay in Sydney. Food and transport is cheaper than New Zealand (better public transport), and houses are better designed for the weather (compared to Wellington / Dunedin) so your energy costs will be lower.

Great standard of living

This is the bit that really rubbed off on me. When you first get here it’s all chromed up bars, bouncers and awful franchised cafes. But Brisbane has some gems, and it has enough gems that once you link them all up you can have a really good quality life here. It’s not as epic a cafe scene as Wellington, and I miss some of the fast food in Melbourne (lord of the fries!), but Brisbane does really well.

The easy long term view

There are as many types of startups as there are types of people, but in the kind of “startup” I find myself working on (part time projects funded by occasional consulting gigs, where the project might turn into a business over time), Brisbane is a really good match. It’s basically a overgrown country town, with all the advantages of affordability and relaxedness, combined with excellent modern infrastructure and a burgeoning creative sector.

Backbone Gotchas

2011-05-17T00:00:00-07:00

If you’re using Backbone and reusing the same div for all of your views (in a single-page iphone app for example), make sure you call .unbind() on the div when you call destroy the view, backbone doesn’t do it for you, and if you don’t unbind the div, you’ll end up with a whole bunch of events (which are using delegate so they hang around) being trigger at strange and inopportune times.

Designing without thinking

2011-05-16T00:00:00-07:00

This is the kind of post which is going to ensure I never get a job with the Aspergers Crowd, but I’m going to put it out there - I sometimes program and design without thinking.

There’s two aspects to this. One is when you’re working on a hard project where the functionality is spread across several files and modules, and you need to do some rearchitecture. I’ll often sit down, work out how I want it to work, then zone out and just refactor without reading the code on the page. Old-school lispers are famous for this, they can work out when to refactor and where the code is unbalanced by looking at the shape of the code, due to all the parentheses. I do a similair thing, relying on syntax highlighting and compile-on-save to point out when I’ve broken something.

The second case is when you’re prototyping a project, and you have to force yourself to not overthink things, to prevent becoming an ‘architecture astronaut’ (abstracting everything away into a GenericFactoryFactoryGenereratorCreator), and instead just deciding ‘I want to to X, so I’ll copy this code here and just search / replace until everything works’. I’ve been doing a bunch of this lately with a backbone mobile project I’m working on. I’ve got about a dozen lines of code that I’m copying and pasting into every model I create. I should create a new base class, and inherit from that, instead of cargo-culting (which is error prone and tedious), but I’m not going to do that until copying/pasting becomes unbearably tedious. By refactoring prematurely, I’m bound to make annoying assumptions that I’ll end having to code around anyway. So don’t overthink it and just copy / paste away until the extraction to make is painfully clear.

Obviously both of these techniques only work in a well tested environment, and I can put aside some time the next day to go over what you did and identify the bits that were left out or done badly.

Oh - and there is obviously that third type of coding without thinking, which is when you need to get something done that you really don’t want to, so you grab a few beers and create variables named the_variable_which_is_a_number_and_gets_incremented.

Proof of work in Javascript

2011-04-28T00:00:00-07:00

Proof of work is the idea of getting a client to do some computational work that is expensive to perform, but cheap to verify.

For example - if you send a client a random number, and then ask the client to keep increasing that number until the md5 hash of the number starts with four zeroes, it will take on average 2^16 attempts to find the matching number. Then the client sends the number back to the server and the server verifies that the hash starts with 0000…

The idea is to make the computation expensive enough that it makes it less worthwhile for spammers to attack your system. It’s a nice alternative to a captcha because it doesn’t require any user intervention.

Example proof-of-work solver

var x = parseInt(Math.random() * 0xFFFFFFFF),
  doWork = function(){
    var count = 100, y, h;
    while(--count > 0){
      x += 1;
      y = x.toString();
      h = hex_md5(y);
      if(h.match(/^0000/)){
        console.log("Solved... " + x);
        return;
      }
    }
    setTimeout(doWork, 25)
  };

doWork();

This code generates a random number, then increases it until a suitable hash is found. The work unit is broken up into 100 tests at a time, with a timeout between subsequent work so that the browser doesnt warn about script timeouts. This runs fine on a modern browser, but I imagine it would nail ie6. For production code you’d probably want to profile the doWork function and adapt the work unit size to keep the browser responsive.

Native solvers

One of the obvious downsides is that any javascript implementation of a hashing function is going to be much slower than a native implementation, so if a spammer wrote a native solver for your proof-of-work, they would be able to trivially solve your work requests.

For example - firefox takes about 8 seconds to solve my proof of work prototype, firefox takes about 6 seconds, and ruby can solve it in 0.23 seconds.

Alternative work algorithms

Many hashes is a well known and provably computationally difficult proof-of-work technique, but given the problem that a browser will never be able to solve them as fast as a modern GPU, it would be worth investigating other work that a browser could solve that could not be solved faster by any other method, for example calculating the size of an HTML layout.

Capt 0.5.3

2011-04-21T00:00:00-07:00

I’ve just pushed capt 0.5.3. Code generation works properly now generate controller and generate model, and the capt watch and capt server commands actually work (I was a bit premature announcing on Tuesday). I’m using this on two projects now and it has been a really nice toolchain.

Autogeneration of spec classes is good for maintainability too. I kinda wish I had been BDD-ing Buddycloud at the moment. That’s some complicated stuff.

Wanted - A local startup

2011-04-21T00:00:00-07:00

I’ve worked on “local” for a really long time, it’s something that’s got under my skin. Whether it’s creating mapping engines, url hierarchy for places or analyzing aggregate user trends, I just really love it. I started a startup in 2006 doing local. We almost got hire-acquired up by Google, but didn’t, long story.

Are there any early stage local startups out there?

Like one or two technical people that are working on something cool and want another hacker on board? Something part time would be a great start for me, I’m happy to toodle along for a while on a cool idea while we work out to get traction.

Things I’m interested in:

Quora for local

People ask ‘where’s a good place to get a beer in wellington?’ and you reply “[Hashigo Zake] and [The Hop Garden]” and the UI pops up little boxes saying ‘Did you mean Hashigo Zake at 25 Taranaki Streeet’? And you click yes and it turns your square brackets into a link.

Reddit for local

People can click on places in a city (reverse geocode to buildings and natural features using a combination of google local search and openstreetmap gazetter) and each place has a discussion board. Going to a city level shows all the conversations going on in the city, sorted by recentness. Build a url scheme like /paris/la-cite/notre-dame. I’ve written a prototype of this.

Delicious for local

Write a bookmarklet that lets you bookmark a webpage, and it hunts out any geodata in the page and saves it to a map for you. You can use this to plan your trips by saving all the interesting looking webpages and then accessing the list of places offline on your iphone. I’ve written this bookmarklet already.

Greplin for local

The user signs in using oauth to foursquare, google my maps, facebook, gowalla and twitter. Index all the places the user has been and all the places their friends go. Use this to generate a ‘personal black book’ of places to go while travelling the world. So you go to berlin, fire up ‘GeoGreplin’ and it shows you all the places your friends want to visit / have checked in to / have saved to a map.

Twitter local browser

Suck down the twitter geo-firehose and plot all the tweets onto maps. Break the tweets down into cities and suburbs. Aggregate the tweets together (using some kind of 2d histogram) to show where the interesting stuff is in your city. People can discover people nearby and see what’s going on. I’ve built a prototype for this too.

Anyway - that’s the interesting ideas. Anyone working on stuff like this? Or any angel list types want to fund me to find a cofounder and build one of these?

There comes a time in every project...

2011-04-20T00:00:00-07:00

There comes a time in every project - after days of prototyping, design, tweaking, rigorous speccing and bdd, after several deployments to staging and a/b testing with beta users, after polls and roundtables and whiteboarding - that you have to admit that the project isn’t any fun to use.

And that’s the opportunity to sit back and reimagine it, then make some small (usually not huge) tweaks until it starts to come together again.

Channel Activity

2011-04-20T00:00:00-07:00

It’s hard to discover new activity in Buddycloud, time to fix that.

The channel list in buddycloud

Log into diaspora-x and there’s no indication of what activity is happening on your channels. Simon said to put the channel list on the left and bubble activity to the top. Let’s try.

span{
  display: inline-block;
  height: 20px;
  width: 20px;
  text-align: center;
  font-size: 0.8em;
  background: #999;
  color: white;
  border-radius: 10px;
}

Okay - add a litte badge showing the number of posts, and reorder the channel list by number of posts.

A sorted list of channels with new activity

Nice.

Diaspora-x ui work

2011-04-19T00:00:00-07:00

I’ve been doing some work on buddycloud / diaspora-x user interface. Screenshots follow:

You have to be friends

I have to admit that the buddycloud ui looks a lot nicer, with blues and colors, but I’ll be sticking with the grey and black ui that diaspora-x features.

The ministry of interesting posts

I’m getting ready to deploy the latest diaspora-x.

Capt is in npm

2011-04-18T00:00:00-07:00

I’ve resumed work on capt, since my diaspora-x codebase relies on capt, and although I expect brunch or cinco will take over as the build tool of choice for coffeescript / backbone apps - capt works for me now.

Logo indicates far more polish than the project has.

Capt is a tool for development and deploying javascript single page apps. It takes care of including all your javascripts in the right order, and compiling your static html pages. It has some support for jasmine, coffeescript and less. It’s very creaky, but my collaborators on the diaspora-x / Buddycloud web client needed to be able to use it, so I’ve packaged up capt as an npm.

So you can install capt with:

npm install capt

Once it’s installed, create a new app and serve it using the built in server.

capt new myproject
cd myproject
capt server
open http://localhost:4000/

And enjoy the capt goodness. I’ve put up a basic webpage explaining how to use capt. I imagine this release is pretty broken, it’s only really intended for diaspora-x development, but give it a go. I’ve been doing small commits on capt for a while now, so it’ll eventually turn into a decent tool I hope.

Coffeescript supersedes Rails

2011-04-15T00:00:00-07:00

This is a bit of a contentious post (that commit is comedy gold) but I think people talking about rails using coffeescript as the new default client-side language are missing the point that coffeescript doesn’t complement rails, it supersedes it.

I come from a biased position, since I’m known for front end development, but I’ve been doing rails since version 0.6, and most of my contract gigs have a ruby on rails component. Rails is a great tool for building middleware, and when it was invented it was a perfect solution.

2006

Common technology was MySQL and PHP. Front end development was pretty low-impact, mostly you generated html on the server and sent it to the client, sometimes you sent html fragment back and forth for some ajax-interactivity.

Into this environment, rails was amazing. It was a great framework for building apps, gave you conventions on how to structure your app, forms and data models. And then rails came out with the ajax helpers and people went crazy for it. Over time rails added excellent json support and people started using it to create pure json api sites (like twitter does these days), but fastforward to 2011…

2011

We now have pure json databases (couchdb, mongodb), localStorage and thousands of public APIs. And any truly modern app will probably be written in pure javascript (and rendering their static html by a node.js instance for disabled browsers). Into this environment, Rails makes less sense. If you’re building a facebook add-in, you can probably do it entirely using FBJS. If you’re using the foursquare api you can do that in the client, same with twitter. If you need to do an app that stores tonnes of data - get a free couchdb instance from couchone. If you need “web scale” - you can use amazon simple db.

Rails doesn’t need to feature in any of this. And although rails will live on forever (like php does), and it’ll probably adapt to it’s new role as an authentication and permissions layer between the client and the database, I would be surprised if a better solution doesn’t come along. It could be a more mature CouchDB authentication / permissioning framework, or Postgres might release a json interface and people go back to writing stored procedures to authenticate / secure requests.

Middleware is becoming less important

I’m sure its just a cyclical thing, we go back and forth between thin client to thick client architectures, but it seems to me that rails might be on the downswing, to be replaced by some hot new tech that takes less work to create pure javascript apps.

Note that I’m not saying node.js is going to replace rails. I think there’s less need for middleware these days.

Planning in the open

2011-04-14T00:00:00-07:00

I’ve been designing today. The current Buddycloud site needs a refresh - so I busted out Jekyll (the tool I use to generate bennolan.com), some css3 and the blueprint css framework.

The in development buddycloud site

It came together quite nicely, and because we’re hosting the repository for the site on github - you too can go visit the site - even though it’s unfinished.

Which brings a dilemma. Do we make the repository private - remove the hosted instance and only require people working with Buddycloud to view the in-development site? There’s sure some interesting / awkward stuff in there - a lot of lorem ipsum text and some rough guesses of price points and a business model.

Well - to be honest, what kills most small businesses (which is what Buddycloud is at the moment) is obscurity or apathy, not giving interested parties too much insight into your business. So we’ll leave it up for now, and then everyone can see the site when it’s ready for ‘public’ deployment.

Localstorage for speed

2011-04-13T00:00:00-07:00

The Buddycloud web client is available on github. I started with the Diaspora-x codebase, and have been backporting in features from the codebase that was started by Stefan Maka (another Buddycloud contributor).

Channel metadata display

Using localStorage to accelerate load time

When you fire up diaspora-x, it takes about 15 seconds to finish loading the #home page. Obviously this isn’t going to play in a world where page loads should take less than 1000ms. Ideally I’d like the page to load in about 100ms. Assuming we can get everything cached locally, that gives a pretty generous time for the browser to cache and run the javascript payload.

The first part of the optimization was to prevent unnecessary requests to the XMPP server. Once you’ve requested all a users posts, there’s no need to re-request the same posts. We can use Result Set Management to only ask for posts newer than the last post we recieved, and then we can use backbone-localStorage to save the posts in localStorage in the current browser.

class ChannelCollection extends Backbone.Collection
  model: Channel

  initialize: ->
    @localStorage = new Store("ChannelCollection")

And then before we start the app…

Channels = new ChannelCollection
Channels.fetch()

Then remember to clear localStorage when the user logs out.

localStorage.clear()

Distributed Social Networking

2011-04-12T00:00:00-07:00

Normally when a client rings up and wants you to build a replacement for Facebook, you smile politely and say that you’re full up for the next 18 months.

But, Simon from Buddycloud has asked me really nicely to build the front end for his distributed social network, so I said I’d give him 5 weeks and see what we can pull off.

Buddycloud

The buddycloud protocol is an extension for XMPP, the system that gtalk uses. Buddycloud takes the existing XMPP extensions (publish / subscribe, rosters, federation) and builds a social network around them. Buddycloud is an opensource protocol, Simon and others are trying to get the protocol in front of the xmpp standards committee.

There are three implementations of buddycloud-compliant servers at the moment, one for ejabberd, one for prosody, and a new one that uses node.js and plugs into any standards-compliant jabber server.

Web front end

Back in December, I created Diaspora-x, which was a first cut at building a social network on top of XMPP. It is a pure HTML5 app, just javascript, html and css - the backend service is provided by the XMPP servers (running ejabberd and prosody at beta.buddycloud.com.

This was an interesting experiment, and showed it is possible to build a responsive social network that gets all it’s data from an xmpp network.

Twitter, Facebook and Buddycloud

It’s interesting to note that new twitter uses the exact same engineering as Diaspora-x, in that when you visit the website, you’re actually loading a large javascript application. The tweets and profile data that you see is transferred afterwards by an XMLHttprequest.

The existing Diaspora-x client (which is the basis for the new Buddycloud web client) does the same thing, it’s a Javascript app, powered by Backbone, jQuery and Underscore.js - which instead of loading tweets from api.twitter.com, it subscribes to an XMPP channel over BOSH (XMPP over http protocol), and messages from your contacts are pushed down to the browser.

The current Facebook UI does a similair thing, it loads a small amount of static html when you load the page, then most of the rest of the page is constructed from Javascript running in the browser. (Facebook is a bit different because it uses their Primer tool that downloads html, instead of doing the templating in the browser - but similar concept).

Building the Buddycloud Web Client

The Buddycloud web client will be an Opensource project, so you’ll be able to watch my progress at github once the repository is up. I’ll try and blog regularly here, so watch for progress…

A* in coffeescript

2011-04-11T00:00:00-07:00

I put together this astar solver in coffeescript. It expects that nodes have a key() function that generates a unique id for the node, and an getAdjacentNodes() function that returns valid adjacent nodes.

It seems to work nicely.

Also - it’s my birthday today! Woo! :)

class AStar
  constructor: ->
    @openNodes = {} # List of openNodes nodes (nodes to be inspected)
    @closedNodes = {} # List of closedNodes nodes (nodes we've already inspected)

    # The maximum potential trip length we would consider
    @maxHeuristic = Point.origin().squareDistanceTo(new Point(5, 5))

  findPath: (start, destination) ->
    # g = 0 #Cost from start to current node
    # h = heuristic(start, destination) #Cost from current node to destination
    # var f = g+h #Cost from start to destination going through the current node

    start.f = @heuristic(start, destination)

    # Push the start node onto the list of openNodes nodes
    # openNodes.push(start) 
    @openNodes[start.key()] = start

    #Keep going while there's nodes in our openNodes list
    while @openNodes
      #Find the best openNodes node (lowest f value)

      #Alternately, you could simply keep the openNodes list sorted by f value lowest to highest,
      #in which case you always use the first node

      node = { f : Infinity }

      for key, n of @openNodes
        if n.f < node.f
          node = n
  
      # No nodes remain in openNodes
      if node.f == Infinity
        # No path could be found...
        console.log "No path could be found"
        return null
        # console.log @closedNodes
  
      # Check if we've reached our destination
      if node.equals(destination)
        path = [destination]
  
        while (node != start) # && (node.parentKey)
          node = @closedNodes[node.parentKey]
          path.push node

        path.reverse()
    
        return path
    
      # Remove the current node from our openNodes list
      delete @openNodes[node.key()]

      # Push it onto the closedNodes list
      @closedNodes[node.key()] = node

      # Expand our current node
      for n in node.getAdjacentNodes() when (!@closedNodes[n.key()]) && (!@openNodes[n.key()]) 
        # console.log(n.key())
        n.f = @heuristic(n, destination)
        n.parentKey = node.key()
    
        # Prevent really long paths
        if n.f < @maxHeuristic
          @openNodes[n.key()] = n
        # else 
        #   @closedNodes[n.key()] = n

  # An A* heurisitic must be admissible, meaning it must never overestimate the
  # distance to the goal. In other words, it must either underestimate or return 
  # exactly the distance to the goal.
  heuristic: (a, b) ->
    a.position.squareDistanceTo(b.position)

Here are the specs so you can see that it does actually work, but the require a bunch of other classes that I’ve been working on - and I don’t have time to tidy them up for release right now:

describe 'astar', ->

  describe 'square map', ->
    # Create a big square tile
    map = new Map
    bounds = new Bounds(Point.origin(), new Point(10,10))
    for point in bounds.getPoints()
      map.set(point, new Tile)
    origin = map.get(Point.origin())

    it "should work horizont", ->
      a = new AStar
      x2 = map.get(new Point(0,5))
      path = a.findPath(origin,x2)
      expect(path.length).toEqual 6

    it "should work vertical", ->
      a = new AStar
      x2 = map.get(new Point(5,0))
      path = a.findPath(origin,x2)
      expect(path.length).toEqual 6

    it "should work diagonal", ->
      a = new AStar
      x2 = map.get(new Point(5,5))
      path = a.findPath(origin,x2)
      expect(path.length).toEqual 11
  
      path = for tile in path
        tile.toString()
    
      expect(path.join(" -> ")).toEqual("0, 0 -> 1, 0 -> 1, 1 -> 2, 1 -> 2, 2 -> 3, 2 -> 3, 3 -> 4, 3 -> 4, 4 -> 5, 4 -> 5, 5")

    it "should several times", ->
      x2 = map.get(new Point(5,5))

      a = new AStar
      path = a.findPath(origin,x2)
      expect(path.length).toEqual 11

      a = new AStar
      path = a.findPath(x2,origin)
      expect(path.length).toEqual 11

      a = new AStar
      path = a.findPath(origin,x2)
      expect(path.length).toEqual 11

      a = new AStar
      path = a.findPath(x2,origin)
      expect(path.length).toEqual 11

      a = new AStar
      path = a.findPath(origin,x2)
      expect(path.length).toEqual 11

      a = new AStar
      path = a.findPath(x2,origin)
      expect(path.length).toEqual 11

    it "should work on a single step", ->
      a = new AStar
      x2 = map.get(new Point(1,0))
      path = a.findPath(origin,x2)
      expect(path.length).toEqual 2

    it "should work on origin->origin", ->
      a = new AStar
      path = a.findPath(origin,origin)
      expect(path.length).toEqual 1

    it "should return empty when impossible", ->
      map.set(new Point(100, 0), new Tile)

      a = new AStar
      x2 = map.get(new Point(100, 0))
      path = a.findPath(origin,x2)
      expect(path).toEqual null

Against me nervous energies sessions

2011-04-09T00:00:00-07:00

For future reference, the against me nervous energies sessions are linked in this post.

My favourite band of all time. Make sure you also check out borne and most importantly, don’t lose touch.

Native maps speed in Javascript

2011-04-08T00:00:00-07:00

Whenever I use OpenLayers or even Tile5, it annoys me that no matter how much you optimize your javascript code, your html5 map will never be as smooth and fluid as the native iphone maps app. I decided to try and fix that today.

This demo is only a proof of concept, but if you try it on an iphone (even my 1st generation iphone), you’ll see that it pans as quickly as the native client. It uses:

A 60fps interval to update the map position
Damped map velocity movement (flicking)
Translate3d to use hardware acceleration
Suppresses touch events and user-select events to prevent the user interaction slowdown
Zepto.js for event management

I’m discussing with Damon from tile5 to get this code put into tile5 (since writing an entire javascript mapping library isn’t on the cards for me today), but I wanted to show the performance that javascript mapping libraries should be aiming for.

Smallest turing complete language

2011-04-06T00:00:00-07:00

I was wondering what the smallest set of opcodes would be for a turing complete language.

Brainfuck seems to indicate that you need 8 opcodes:

Move the data pointer left or right
Increment / decrement the data pointer
Write the data pointer to the output
Read input to the data pointer
Loop until the data pointer is zero

AI smallest program

2011-04-05T00:00:00-07:00

I’ve been thinking about AI a bit lately, and a question I’ve had, is what is the smallest non trivial programming loop you could make? What number of opcodes would it be? I guess it’d have to be a program that took evolving inputs and iterated to approach an optimal solution.

I’m thinking of a program that has changing inputs, so that there is no one optimal solution, the best the program can do is constantly seek to a solution. It’s a thought experiment, in that that you can calculate an upper bound for how quickly a genetic program could evolve by working out the simplest non trivial program, and calculating how fast it could be run on a modern cpu.

Simple English

2011-04-04T00:00:00-07:00

Simple english has only 800 words, and I’ve found an english -> basic english translator. I think it’s a good vocabulary for a chatbot.

Hypergraphs

I’ve been reading about hypergraphs, which is what OpenCog uses as a knowledge store. It’s a graph where instead of edges being between nodes, edges can have have an infinite number of nodes. You could look at it as a ‘set store’, where you have many many sets of different ‘strengths’, surrounding a fixed pool of nodes. Anyway, I’m not totally sure I understand the theory, but it seems like an interesting idea.

My naive implementation of a knowledge store for a chatbot, is to create nodes for terms, and then have bidirectional relationships between terms, and relationships are strengthened when the analyser is fed two terms in sequence. As far as i can tell, this is like the hypergraphs.

Metropolis-Hasting

After working on Indigo Renderer and having many conversations with Nick Chapman, I have a sense that when doing a random walk through the AI’s knowledgebase, that it’d be worth calculating a probability that the path would have been selected.

So for example - the AI queries it’s knowledge store for the term ‘dog’, and the knowledge store would query for a thousand random chains of related terms from dog. It would then rank those chains in terms of likelihood of those chains having been taught to the machine.

So - train the bot:

dog is animal
dog is pet
horse is animal

And the knowledge store might return:

0.2 dog is animal
0.2 dog is pet
0.15 dog is animal is pet
0.1 dog is animal is
0.1 dog is pet is
0.1 dog is pet is dog
0.01 dog is animal is horse

Brunch

2011-03-23T00:00:00-07:00

I’ve stopped developing capt, my javascript build tool. I’m instead contributing to brunch, a tool built by Jan and Thomas. It’s still early stages, but it is already ahead of where capt was. My todo list for brunch is:

Add jasmine testing to clients
Port static html generation templates from capt
Add code generation for classes / models

It’s an interesting project, hopefully a good alternative to the upcoming framework from @sstephenson.

The golden ratio in CSS

2011-03-22T00:00:00-07:00

I’ve been designing a new mapping interface for a product I’m releasing shortly. I’ve copied the design of the twitter app for ipad, which has multiple sliding panes. I was trying to work out the widths of each pane so that the app looked a bit more cohesive, when I decided to try out the golden ratio.

I’m using stylus - which is an excellent css toolkit, similar to sass or less.

golden-ratio = 1.61803399
base-width = 48px

The base-width is the width of the skinniest column on the left side of the page. I then use multiples of the golden-ratio and the base-width to chose the widths of the other columns.

.modes-presenter
  width base-width

.places-list-pane
  width (golden-ratio ** 4 * base-width) - 40px

.place-show-presenter
  width (golden-ratio ** 4 * base-width) - 40px

I’m not sure if it was a placebo effect - but the end result was quite pleasing to me.

Early prototype with GR-sized columns

Back from holiday

2011-03-21T00:00:00-07:00

On the 1st of February, Riss and I moved out of our apartment on Mount Victoria, and into our little green van. We had a bed in the back, gas bbq, chillibin and a box of food. We spent the next 7 weeks travelling around the South and North islands of New Zealand.

Totaranui, Golden Bay. Beautiful place.

I had created a rough map of where to go using a beta version of Weheartplaces, but in the end we just travelled using the lonely planet and my ipad to look up places on the go. As it was, it turned out that a place bookmarking service would’ve been really awesome, especially in finding freedom camping sites, as they are scattered throughout the internet and hard to find.

(As it happens, I’m working on a solution on finding freedom camping sites as we speak).

Camping in New Zealand is beautiful. The doc campsites are the best places to go. They’re remote, cheap, relaxed and have the best locations. My favourite three places were Totaranui, Whananaka North and Aoraki campgrounds. I’m still working on culling and uploading all the photos from the trip - but it was a very fun time - and it’s great to be back.

Gear review:

Nissan 4wd Van - excellent. Big enough to sleep in (although we had a bit too much stuff), small enough to get into all the great little campsites. 4wd was good for rallying up.

iPad w/ 3G - excellent. Hugely useful, but not so capable that I could do any more than briefly reply to emails (can’t do any coding on it, which is good). Vodafone coverage is fucking awful, telecom 3g is much better.

Chilli bin - good. Buy some ice every few days and you’ve got a cheap, easy to use fridge. Only thing missing was it didnt have a tap at the bottom to let out the melted water, and it couldnt fit heaps of beers.

Gas cooker - excellent. You need a twin burner, so you can make fancy meals. We ate really well, and making coffee with the burr grinder and stovetop espresso, while sitting on a remote beach, is really awesome.

Desaturated google maps

2011-01-27T00:00:00-08:00

You’re building your site, you do it nice, you have regular line spacing, lots of white space, even gaps, nicely kerned typography. You took your triplet css values #ccc to #c3c3c3 for that perfect shade of grey, changed all the borders from dotted to solid. Your website is maybe starting to look like the pages of a design magazine. Then you add google maps to the page.

It’s like smearing playdough on your furniture.

Try using google styled maps and desaturate the map to get something that looks more like this. Still functional, but easier on the eye and suitable to my bauhaus sensibilities.

Desaturated, label opacity reduced

Fix myspace in 5 steps

2011-01-25T00:00:00-08:00

I have a bit of a thing for trying to fix broken social networks (see Diaspora-x), so when I recieved an invite to friend someone on Myspace, it jogged my memory that the company is up for sale, so I thought I’d make my list of 5 things the new owner could to to stem the decline and maybe even compete with the #1 social network.

Become a flat engineer-oriented organisation

From reading this reddit iama, it seems that the engineering teams aren’t communicating effectively. Given the glacial speed of improvements to the site, I imagine that the development process is pure corporate-bullshit, where any improvement or tweak must go up through a layer of bureaucracy before being built.

Instead, let engineers discuss a feature with their team lead, build it and deploy it in the same day. The same way facebook does. And make the default answer to be ‘yes do it’, not - ‘uhh lets think about that’. Companies fail because the employees can’t innovate, not because the employees fuck up.

Become design and user experience focussed

The myspace user experience is poor when compared to Facebook. It’s hard to tell where you are in the site, it’s hard to find information, it’s hard to work out how to do simple tasks. The biggest things that need fixing with the UI right now:

Myspace style guide. All myspace pages, from the developer guide to the front page to the sitewide UI should obey the same style guidelines for consistency.
Focus on page load time and browser stability. Deprecate flash, promote and develop html5 solutions for the current widgets. Don’t autoplay any media.
Add summaries of profile sub pages on the main profile page (show top 3 photo thumbnails, links to the top 3 tracks on the music page, create a summary field that shows the user / artists most recent activity) to encourage users to explore deeper into the profile and not just hit ‘back’.
Add more explorability. Make record label links go to a ‘label page’ that shows other artists. Make the ‘location’ field a hyperlink that shows a page with all public profiles located in that area.
Hire pixel-perfect designers. You don’t have the facebook ui template library, so make developers forward a screenshot of any new UI (be it a ‘you are logged in’ alert or a ‘changes saved’ interstitial) to your design team, and give the design team a mandate to sit in with developers and show them how the pixels should be moved around until everything lines up.

Galloping baseline, overlapping elements, multiple colors, inconsistent spacing, no avatar image.

Become more open

Myspace should move their APIs over to the opengraph and do a staged deprecation of their current APIs. The focus should be on letting third party developers access the myspace data. Creative commons licences should be promoted and liberal licences should be granted to myspace data. Users should have control over their privacy, but anything that has been elected to be public should be shared freely and instantly, no more summary-only rss feeds, no obfuscated URLS, all public content should be freely and easily available.

Make better tools for bands

The tools for bands to edit their tour schedule, keep in contact with the fans or maintain their page should have a UI overhaul. A nice, simple, ajaxified design that lets them get in, update their tour and music and then get out again. The bands data should be made available through APIs if the band decides too - so that myspace becomes the central location they manage their social media presence. For example, their myspace page should have a podcast url, their tour data should be syndicated out to upcoming, last.fm, they should be able to set creative commons (or other) licences on all their media.

Start a dialog with your community

Most people think that Myspace is fucked. You need to start communicating on the myspace blog, often and always, and show the work that is being done to make Myspace a better place. You need to point out where Facebook fucked up and you did it right. You need to be seen to respond to users concerns about spam, about it being hard to delete profiles. You need to post screenshots of the new nicer user interface you’re working on (here’s the new inbox! here’s a video of how the profile editor works).

Finally, I don’t know how the MySpace revenue model works, but they should invest in their advertising technology so that they can provide the best possible solution to their existing advertisers and attract new ones. The site needs to be profitable and strong on it’s own account and that should be by high quality, targetted and effective advertising that doesn’t detract from the user experience of the site.

So that’s my thoughts on building an a more agile, more profitable and more enjoyable Myspace. Competition is good and MS has the best opportunity of sticking it to FB.

Thoughts from someone leaving NZ

2011-01-24T00:00:00-08:00

Since my girlfriend and I are leaving New Zealand for good probably, I decided I’d write about the New Zealand economy from my opinion as a 30 year old. Take this with a grain of salt, as it is purely anecdotal, I don’t have time to lookup citations for my observations.

Me

I’ve started two business in New Zealand. One, Indigo, generated about 100k€ of export revenue in the first year, employed 5 staff and is growing faster than ever. My co-founder has just left New Zealand with the rest of the company so they can be closer to their export markets in Europe. My other business, Zoomin, employed a half dozen people and was part of the Wellington startup cluster that jumpstarted in 2006. Rissa and I are now off travelling around New Zealand in a van, then moving to Brisbane for a few months before travelling to either the US or Europe.

It’s expensive here

Food and entertainment are expensive. It’s cheaper to shop at a Tengelmann supermarket in Munich using New Zealand dollars, than it is to shop at a New Zealand supermarket in New Zealand dollars. New Zealand cheese is cheaper in Australian supermarkets than in New Zealand supermarkets. It’s interesting talking to European backpackers who are astounded at how expensive it is to travel around in New Zealand. $42 / night for a campground? $8 for sausages? $5 for a loaf of bread? $18 for a six-pack of beer?

Housing is expensive here

As an investment, I feel uncomfortable taking on an extra two years worth of salary as debt, when you don’t need to take that much debt to buy a house in another country. It seems especially perverse because there is so much good available land that is undeveloped. We’re not Japan or Germany.

The economy is overinvested in property and finance

The UK learnt that relying purely on investment in property and the financial markets wasn’t a good idea over the past decade. They look at producing countries (like Germany, Japan and China) and realise that there is actual value in inventing technology, processes and processing businesses. New Zealand really only invests in the primary market and property. There are some well known exporters like Xero, but they’re about 0.01% of our industry. And New Zealand doesn’t seem to have realized yet that they need to become a strong economy in secondary industries (like processing, manufacturing, designing).

New Zealand should have a distance discount

Being so far from our customers, investors and partners, people should have a discount to work in New Zealand. Instead it’s more expensive to live in Wellington than to live in Munich.

It’s lovely here

I used to hate on New Zealand a bit much and I’m sorry for that. It’s really a beautiful country. The people are nice, the coffee is awesome. When things are slow, affordable and easy this is a wonderful country. The diving, beaches and mountain biking are off the hook.

We’ll be back

I hope that we’ll be back in New Zealand to live, or at least for our holidays. My prediction is that the property market will stagnate for half a decade, while salaries slowly creep up to bring affordability back. This relies on the government taking steps to squash investment property (since investment property tax advantages make an investment property a more affordable purchase than a first home).

Affordability is good for everyone

Part of the problem with the gap increasing between rich and poor, with people having to spend 120% of their salary to pay a mortgage, is that people are stressed out. I don’t want to live on a street full of renters who live in cold, substandard housing. I don’t want to live in a street of people who are working all hours and stressed out to meet their repayments.

The labour government was atrocious

I’m a Greens supporter, so in theory I should support Labour more than National, but Labour did absolutely nothing to protect New Zealands quality of life from the stupidity of the consumer debt bubble. It was John Key who did the first (half assed and limp wristed) attempts to redirect investment from the property sector into industry.

In a perfect (unlikely) world, property investment would be limited purely to apartment complexes and fresh developments in subdivisions, and tax-breaks would be geared to encourage owner-occupiers in existing properties.

Let’s see what happens.

Heroku or Linode

2011-01-20T00:00:00-08:00

The biggest problem I have with keeping my projects going is paying for and maintaining my servers. When you’re consulting, $60/mo for a dedicated server isn’t a big deal, but when it’s seeing no real traffic growth and you have to be responsible for backups, updates and admin - I find that after 3-4 months I usually mothball a project and have one less thing to administer.

Thus Twitterplaces, which was running on a dual-core 4gb box so it could index all the geo-tweets in the world. I left it run for about 6 months before taking it down. I’m planning on releasing weheartplaces to the appstore before I head off on holiday - so I need to set up a production environment for it. My options as far as I can tell are:

Heroku

Pros: Managed rails hosting, don’t have to do any admin, everything is backed up.

Cons: Expensive (minimum of $50/month), no postgis.

Linode

Pros: Affordable ($20/month), can install whatever I like (Postgis / R / FastCGI scripts).

Cons: Have to sysadmin and back it up myself.

I’m leaning towards linode at the moment since I already have one running diaspora-x.com, but I’d like some feedback on this. I know that at some level I should put my app on Heroku - but maybe I can have it running on my 512mb linode for a few months while I see if there is any uptake in weheartplaces at all?

Quantitive measurement of bus drivers

2011-01-19T00:00:00-08:00

I live in Wellington, New Zealand - and we’re unlikely to get modern tramways for a while, but we do have an extensive bus network. Busses are pretty average compared to trams, but I don’t think it has to be that way, and I’d like to propose some quantitive measurements that can be done to ensure the quality of drivers of the busses.

The main trick is to install a device in each bus that records the bus driver, the route, the timeliness of the route, comfort and fuel efficiency. You could use something like an android or iphone, since they have gps, accelerometers, a user interface and 3g connectivity.

Bus driver

The bus driver gets on the bus, types their name into the phone and they are logged in. They then specify which route they are running and the measurement system can work out where the bus should be.

Timeliness

Measure the location of the bus against the measured stops. Use this data to show a timeliness report (as they do in Melbourne) and use this data to decide what changes need to be made to routes, traffic lights and bus lanes. Instead of drivers having to rely on anecdotal evidence, they can show why changes need to be made to each route, for example, a sector may need to be made longer or shorter, route timings changed.

This could be done on an ongoing basis as traffic patterns change season by season.

Comfort

This is a big one. The major problem with busses after timeliness, are bad drivers. The main factors of a bad driver:

Accelerate really hard, causing a huge amount of noise. If you’ve ever seen a baby sitting in a pram on lambton quay when one of the old diesel busses accelerates away you’ll know what I mean. This is uncomfortable for passengers, adds unneeded noise to the city, is dangerous for people on the street, promotes aggressive driving, and wastes fuel.
Drive too fast through corners. Some bus drivers drive their busses like race cars. And some driver their busses really carefully and considerately. The considerate drivers need to be rewarded.
Break too hard into a corner, don’t predict the lights and coast to a stop.

By measuring the gps location and instantaneous acceleration 10 times a second, with the phone, or measurement device, mounted in a rigid cradle at the front of the bus, it would be possible to create a measurement for how smooth and efficient a bus driver was.

It would be easy to measure when a bus driver wasn’t driving smoothly and comfortable.

Learning and improving

I’m not sure what the best way to use this data would be. It seems like paying bonuses or penalizing bad drivers would be the most effective way to ensure a competition for smoothest driver, but I imagine there are kinder ways of rewarding correct behaviour.

It would be very important for all measurements to be compared correctly against other drivers on the same route, on the same bus, and at the same time of day, as some routes would be by necessity more intense than others. If you’ve taken the bus through vogeltown or roseneath you’ll know it’s a pretty windy path.

It would also be important for the data to be used to make changes at the roading and management level as well, instead of just passing the buck onto bus drivers and expecting them to do miracles. Wellington City Council should improve busways where needed, relying on real data, not anecdotes. And the bus company management should be made accountable as well.

Especially for buying the bad old MAN busses that are wheelchair-hostile and noisy as hell.

Public data

Finally, all this data should be publically available (in an suitably anonymized, aggregate form) so that ratepayers know that they are getting value for money from their subsidised bus service.

(If anyone actually wants this thing to be built and exist, I’d recommend getting a local company like youdo, southgate or silverstripe to build it)

Weheartiphone

2011-01-18T00:00:00-08:00

I got inspired by the design of plaintext and redesigned my phonegap app. Weheartplaces is an app for bookmarking places and accessing them on the go. It syncs with your facebook account so that in the future you’ll be able to see your friends bookmarks and send bookmarks to friends.

Koala for facebook

2011-01-17T00:00:00-08:00

I’ve been using Koala + Facebook connect for facebook development in rails. It’s an awesome tool for exploring the graph API using pure rails. You don’t need to use devise to do your authentication, just facebook connect, then take the access token and then you have access to the facebook api as follows:

@oauth = Koala::Facebook::OAuth.new(app_id, code, callback_url)
@oauth.get_user_info_from_cookies(cookies)

It’s interesting when deciding where to use APIs like facebook or the google maps geocoder api. Should you do it client side, and reduce load on your server, but add the complexity of handling state and large amounts of data in an environment that’s not really designed for it - or do you do it client side and increase your ops cost?

It’s still a hairy one I haven’t decided. One of the classic problems is when people search on a social mapping site (like weheartplaces):

/places/search?query=Eiffel+Tower
/places/search?query=Nelson,+New+Zealand

You want to look for any places called eiffel tower in your database (probably using tsearch2), but you also want to run the search through a geocoder to see if the user actually searched for a city. And you need to do some kind of combinatorial weighting function to work out what was the most likely result for a given query.

The best way to do that is to gather data on the server side so that you can train your weighting function based on previous results and what the user was actually looking for.

Pure javascript and focus on the UI? Or a strong back-end service and focus on the data quality? Or both? And never finish the project.

Capt in production

2011-01-14T00:00:00-08:00

I’ve added some features to capt, it’s still very rough and needs a lot more work, but it’s coming together. Added recently:

Build command

Run capt build iphone to create a build in builds/iphone that will compile all coffeescript, concatenate and minify all scripts, compile all the less into css and run the yui compressor over it, then create an index.html from your index.jst, with only two files as an include.

Added support for targets in the config.yml

I have this on my config.yml:

iphone:
  lib/phonegap.js
  platforms/iphone.coffee

These are included only when developing against the iphone, and are all merged into one file for production. I do the same for nokia (which requires json.js and some special code to deal with the nokia webruntime).

More test support

I’ve improved the testing templates, but I still need to get a command line test environment running - basically type capt test and have it run the whole test suite using node.js and jsdom.

Future of capt

I doubt the first decent release of capt will be prior to April, since I’m heading away on a campervan holiday of new zealand for 2 months, but keep an eye out then, and feel free to fork and send pull requests in the meantime.

Jquery Mobile or not

2011-01-13T00:00:00-08:00

I’ve been working on weheartplaces again, trying to get it into the appstores and out the door. I used capt to build the bundled and compressed javascript, and using jquery mobile, the closure compiled code was 161k.

I removed jquery and jquery mobile and replaced it with zepto.js. It took about an hour or so to recode a minimal CSS theme that replaces all the jquery mobile functionality (with the exception of fixed position viewports) and the resulting closure compiled code was 41k.

The huge difference was in application startup time, and in less crashes on the actual device. On the simulator, the javascript compiles instantly, but on my 2g iphone (same as a 3g iphone processor), the app starts in 4 seconds, instead of the previous 8. It’s also smoother page to page and the user interactivity is quicker.

jQuery mobile is fantastic and I may end up going back to it - but for now I’m going to do my mobile development using jquery.js (zepto doesn’t work on Nokia), backbone.js and a custom stylesheet. I’ll put up some screenshots of the new weheartplaces mobile app shortly.

2011 - the year of pure javascript apps

2011-01-12T00:00:00-08:00

Pure javascript apps, or apps that use javascript for all the presentation have been coming to prominence for years.

Google maps 2007
Phonegap 2008
Facebook 2009
Twitter 2010

It’s an obvious prediction, but 2011 will probably have a lot more javascript apps being released. From mobile - where phonegap is so important to smaller companies that can’t afford to redevelop an app natively in Java and Objective C - through to ‘desktop’ web apps like the twitter client.

Some of the key technologies that I peg for uptake this year:

CoffeeScript

I’m a big fan of this. It’s easy to learn (if you’re open minded and don’t try to be too clever), and makes javascript immeasurably more pleasurable to write. It’s basically a whitespace aware javascript compiler, so it does away with all the curly brackets, replaces function with -> and has very nice built in prototype-based class support.

CouchDB

This one I’m not so sure about. Couch is really amazing for creating pure javascript apps, since you don’t require a middleware, you can put business logic into the database layer (rules are written in javascript) and they talk JSON to your javascript app. The downsides with couch are that it is a bit of a mindfuck to get used to, coming from a relational background. Also, the couchapps / authentication / client-facing parts of couch are relatively new and I think there needs to be a lot of work on documentation and frameworks to make client-facing couch apps easier. So, couch for indie developers making pure javascript apps - watch out for this.

Backbone.js

There are a dozen or so javascript MVC frameworks. I imagine they’ll all take off (javascriptmvc, backbone, ext..), but the one I really like is backbone. It’s pure javascript, plays nicely with Zepto (a tiny implementation of jquery for webkit-mobile) and works well with coffeescript. It’s very event based, so some parts are really great (like redraweing the UI when an item in a collection changes), but it’ll be interesting to see how it progresses over the year, hopefully some great tutorials come out of it and it’ll become a dominant force in javascript development.

Phonegap

I know that mobile apps written in phonegap are seldom as good as native apps, and a well written native app will always be better than an html+css app, but phonegap is just going to get better and better this year. If you look through the phonegap plugins for each platform, there are some great bits, like the email connector and child browsers.

Node.js

To finish off, node.js. Node is a bit of a mixed bad. It’s awesome because everything is async, it compiles easily, is very fast and seems to be reliable. On the downside, the API changes like a bitch, constantly. Any code you wrote for node 6 weeks ago, probably won’t work anymore. I appreciate that it’s a pre 1.0 project, and they’re trying to work out how to do everything in the best possible way, but if development really needs to stabilise around some core areas soon (like socket programming).

CouchDB OAuth

2011-01-11T00:00:00-08:00

I think couchdb is a good solution for creating pure javascript apps, but there are a few big problems with using a service like clouddb, particulary around authentication. There already exists a couchdb oauth module, but afaict, it’s for authenticating against couchdb, not for creating a couchdb account against a twitter or facebook account.

If someone like couchone added a simple way to implement an oauth consumer in their apps, it’d be a really fantastic way to prototype apps. You could authenticate using javascript against twitter, google or facebook - create the users account and database, and then return them to your application, all without writing any server side code.

The current couchdb authentication is a bit basic and doesn’t support password reset or email verification, and you can’t really build such things as couch apps.

Couchdb behind Apache

2011-01-10T00:00:00-08:00

I had some issues getting couchdb to work correctly behind Apache. The big problem I was getting was that cross-domain requests were being “preflighted” where the browser sends an OPTIONS request to the server, which was being proxied to Couchdb - which was returning error 405, since couch doesn’t support the OPTIONS method.

Here is the complete virtualhost setup for giving your localhost apps cross-domain access to couchone.

<VirtualHost *:80>
        ServerName mycouch.com

        Header set Access-Control-Allow-Origin "*"
        Header set Access-Control-Allow-Headers Content-Type
        Header set Access-Control-Allow-Methods "GET, PUT, OPTIONS, DELETE, POST"
        Header set Access-Control-Max-Age 3600

        ProxyPass / http://mycouch.couchone.com/ nocanon
        ProxyPassReverse / http://mycouch.couchone.com/

        RewriteEngine On
        RewriteCond %{REQUEST_METHOD} ^OPTIONS
        RewriteRule .* /index.html [L]
</VirtualHost>

Backbone and couchdb

2011-01-07T00:00:00-08:00

I’ve been using the backbone-couchdb connector by Jan and I’ve been impressed. The real showstopper is realtime updates using the changes feed from couchdb.

Background

For complex geospatial apps, you can’t go past rails + postgis as a toolchain, but for simpler apps where you do all of the work on the front end and basically need a hash store, rails is overkill.

Instead of having to run rake migrate everytime you want to add a column, you just write to the backbone model, call save and it’s serialized away into your document store.

Schemaless stores

It’s possible to use postgres in a schemaless way by either:

Autogenerating columns as they are referred to
Serializing all the columns into a blob

But it makes sense to investigate databases that are designed for the purpose. The two I have experience with are:

MongoDB
CouchDB

Both are good solutions, I’ve used MongoDB extensively with node and it’s json support is very good. I was interested in trying out couch since it is an almost pure-javascript solution, with javascript views and support for couchapps.

CouchDB also runs well without any middleware, your app talks directly to the database.

Backbone-couchdb

Backbone, if you haven’t seen it - is an excellent tool to give structure to your javascript applications. I use it extensively. One of the great benefits of backbone is the use of models and collections, both of which emit events when their contents change.

So say you have a list of posts on the left side of your page, you bind your render() (or redraw) function to the add and remove events of the Posts collection and the list will always be up-to-date. For example - if you opened your javascript console and typed:

Posts.add(new Post(content : "Hello world!"))

All the dom elements will be automatically created and the posts list redrawn.

Backbone by default doesn’t come with any persistence layer. Models and collections are stored as javascript variables and they are lost when the page is refreshed. You can persist using the build-in sync methods that submits json to a server. Backbone uses the rails idioms for this, although it’s easy enough to update Backbone.sync to work with other datastores / application frameworks.

You can also use the localStorage plugin to persist your models to the browser. The backbone-couchdb persists your models to the couchdb store, so whenever you call save, it posts your model to the server - and when you reload the page, it loads all your models from the server.

And because it’s couchdb, all you have to do is create the database record in futon (one click) and all the models are automatically saved away, the database doesn’t need to be aware of what you’re saving, it just saves.

The magic of realtime

The magic part of this is the _changes feed from couchdb. A client can request the _changes feed by doing an ajax request. Couchdb will then block (ala long polling) until there are changes on the database, at which point it will send down json and close the connection. The client processes these changes (updating collections and models as needed) and then reopens the connection to the changes feed.

And because you are already re-rendering your UI whenever a collection changes, you can see in realtime what anyone else is doing on the same dataset that you are using.

Couchone

I’m using couchone to host my couch databases. It’s a great service, the only downside is that you can’t set Access-Control-Allow-Origin headers, which would let you do cross-domain ajax calls (so you can host your app at github and store your database at couchone). I got around this by proxing through my linode box, but this kills performance. Hopefully the couchone guys will give you the option to change the headers (they could use varnish or similar).

Diaspora-X speedups

2010-12-30T00:00:00-08:00

I’m back from a week on the Australian beaches (I recommend surf lessons at Coolum and the breaks down at Burleigh heads) and at a wicked little festival (Blah blah blah is a highly recommended party), so it’s time to fix up a few issues that were raised while I was away.

About two weeks ago I wrote Diaspora x², a port of the Diaspora UI to pure Javascript. My original Diaspora-xmpp port used rails and xmpp4r to federate by XMPP. It was a good idea, but I’m much more at home using javascript that rails, and there was a lot of work required to get the xmpp connector working reliably at high loads.

That’s why I got strophe.js, which is a javascript library for doing xmpp over the http transport, bosh. The result was diaspora-x2, which is now online at diaspora-x.com.

All well and good - but the first release was REALLY slow. Todays fixups:

Queue rendering calls

My views watch the posts collection for add and remove events. So an xmpp message comes in, it gets added to the posts collection, and then the view re-renders. The problem is, the way bosh batches messages together means that I often iterate over a 100+ messages, and add them one by one to the collection. This then triggers a 100+ render() calls, each of which constructs a big string, the string is parsed, then the elements are added to the dom, then they’re all deleted and it starts again.

Super easy fix that it applicable to a lot of backbone apps:

render: ->
  @el.html(@template())

Replace with:

render: ->
  if @renderTimeout
    clearTimeout @renderTimeout
  
  @renderTimeout = setTimeout( =>
    @el.html(@template())
  , 50)

The only downside to this is that you have to make your view tests async.

Better rendering

The bigger problem is that because the entire posts wall gets re-rendered everytime theres an update means that anything you were typing into an inline textbox will get annihilated. I’ll be fixing this by progressively inserting and removing dom elements when the collection changes. Ordering will be done by having a data-created-at attribute on the elements, so you select the dom elements, then move down the list until you get to the correct element and insert.

Embed.ly

This will let me re-enable embed.ly, so that we can have cool inline media, like soundcloud tracks, youtube videos or flickr photos.

Media upload

I want to enable drag and drop media sharing, where the user selects which service they want to host the media. By default we could support imgur, then the user drag/drops images up and they get uploaded to imgur and then post the text link.

A good cofounder

2010-12-19T00:00:00-08:00

I’ve had the good fortune to have had a co-founder in both of the startups I’ve been involved with. Given the option, I would always elect to start a business with someone else.

However - before you meet your next cofounder, there will be a period of time where you work on your own idea. Before I founded Zoomin (which we almost, but not quite, sold to a multinational), I worked on the prototype (a social mapping site) for about 4 months.

In a lot of projects, you will have to be a one man band for a while. You’ll have to promote yourself, develop the software, motivate yourself everyone morning.

From prior experience, I know that I can do all of these:

Talk to people and investigate the market (research)
Code, architect and design (development)
Sysadmin, keep records and pay taxes (operations)
“Do marketing” and write blog posts (promotion)
Set a price and ensure people pay (revenue)

The things is… I don’t think anyone who tries to build a business ever realizes how hard it is going to be - before they set out.

This week

2010-12-17T00:00:00-08:00

It’s time to drink a beer and write a blog post. This week I worked on diaspora-x squared, an xmpp client written using javascript. It uses strophe.js and backbone.js. I had a few issues that needed attention with backbone:

Multiple render calls

I was receiving xmpp stanzas and inserting them into a backbone collection. The collection was firing the add event and I had hooked that to call this.render. That meant that I was calling render (and the browser was parsing html, creating dom nodes and reflowing the layout) about 30 times in a row, which is fucking inefficient (even though the browser never rendered the nodes).

This could be fixed by putting in a 20ms timeout before calling this.render, and canceling the timeout when a subsequent update request came in (you need to make the timeouts at least 20ms or some browsers will call the timeout immediately).

LocalStorage

I didn’t implement localStorage to cache the previously received xmpp stanzas, mainly because there is no localStorage support built into backbone. I know that backbone is meant to stay lightweight and tiny, but it would be nice if the localStorage store was “officially supported”, then I’d use it more often.

Confusion about submitting forms

When I write form submission code, I do it in a view - so the view has a submit method which is hooked using the delegateEvents method on form submission. I end up processing the form entry, saving the model, adding it to a collection and then doing a redirect (using window.location.hash), all in the view. Some of this code feels like it should be in the controller, but I’m not sure how to do that. Maybe I’m just getting confused with the rails way of doing things.

Adding production support to capt

I’ve used capt (my code generation / development mode server tool for creating backbone apps) to build two apps now and it really is a good tool. I need to add:

Production mode (concatenate files in order, run through closure)
Server side coffeescript processing (not really needed, but a nice to have, esp. with growl support)
Working test framework (I am generating test cases but haven’t been running them, need an easier way to do so)
Documentation and promotion (so that people learn about it)

Interesting week all the same. I had two really good days in the flow (2000 loc on thursday, about 800 loc today). Invented two nice things, an xmpp-pubsub social network (diaspora-x2) and a bookmarklet that scrapes geodata from a currently open page (for adding places to weheartplaces).

Two levels deep

In my weheartplaces bookmarklet, I use this bit of code to search the dom for any google maps instances in the first two layers of the dom, and then extract the lat/long of the map and use it for the bookmarklet:

if google? and google.maps? and google.maps.Map
  maps = []

  # Search two levels deep...

  for k,v of window when v instanceof Object
    if v instanceof google.maps.Map
      maps.push v
    else
      for i,j of v when j instanceof Object
        if j instanceof google.maps.Map
          maps.push j

  for map in maps when map.getCenter
    try
      attributes.geo.push [map.getCenter().lat(), map.getCenter().lng()].join(", ")    
    catch e
      # ...

  for map in maps when map.getCenterLatLng
    try
      attributes.geo.push [map.getCenterLatLng().lat(), map.getCenterLatLng().lng()].join(", ")    
    catch e
      # ...

Works really well. Quite stoked. I’ll try and release the weheartplaces bookmarklet next week.

Attacked by wheelchair!

2010-12-14T00:00:00-08:00

Bwahahaha the bbc. The world media is really doing a great job of acting like fucks at the moment. My favourite quotes from this video: “There are allegations you were rolling your wheelchair towards the police” and (to the cerebal palsy sufferer) “There were people throwing chunks of concrete at the police. Were you throwing things at the police”.

Those riot police do earn their money though. Just imagine standing there, in the open, having a guy in a wheelchair rolling towards you.

Fucking terrifying.

To be honest, the police did have riot gear to protect themselves.

And there were hundreds of them.

But the dude had a FUCKING WHEELCHAIR!

Those wheelchairs are pretty much a deadly weapon.

Capt, a tool for backbone.js

2010-12-13T00:00:00-08:00

I’ve been working on capt for a few days now, so I wanted to write an introductory post. Capt is a tool for generating and serving backbone.js projects in development mode. It’s currently tooled for my personal toolchain:

jQuery
Backbone.js
Underscore.js
QUnit

You can imagine capt as a combination of jamit and the rails code generators. For example:

 $ capt new blogproject
 * Creating folders
 * Downloading libraries

Create a new project

$ find blogproject
blogproject
blogproject/app
blogproject/app/controllers
blogproject/app/models
blogproject/app/views
blogproject/app/views/jst
blogproject/config.yml
blogproject/index.jst
blogproject/lib
blogproject/lib/backbone.js
blogproject/lib/coffeescript.js
blogproject/lib/jquery.js
blogproject/lib/underscore.js
blogproject/public
blogproject/public/stylesheets
blogproject/test
blogproject/test/controllers
blogproject/test/fixtures
blogproject/test/models
blogproject/test/qunit.css
blogproject/test/qunit.js
blogproject/test/views

File structure

$ capt generate model post
Created app/models/post.coffee
Created test/models/post.coffee
Created test/fixtures/post.yml

$ capt generate model comment
Created app/models/comment.coffee
Created test/models/comment.coffee
Created test/fixtures/comment.yml

Create models and tests

$ capt server
node-router server instance at http://*:3000/

Start a webserver in production mode

Capt is written in coffeescript and uses node.js (I’ll distribute it by npm once it’s done), but at the moment is just for client work, it does server side processing to automatically include the correct files, but you’re not meant to develop node apps with it, it’s for developeing client apps.

You can then browse to /index.html and it’ll automatically include all the correct files in the order specified in config.yml. You can also browse to /test/ and it’ll run the auto generated tests.

The default generated tests running in qunit

It’ll be a while until there’s a good first release (I might take the current random code and covert it to use Cake for example), but here’s a headsup if anyone it working on something similar.

Production targets and textmate

Once I’ve got things going - I want to have a few production targets:

Nokia WRT
Phonegap
HTML5 standalone w/ manifest
Web deployment

With a one line command to package up the project into one closure-ified project. I also want to create a textmate bundle for running the tests from within textmate by hitting command+r - like you can do with rspec. I’d like the tests to run within the node jsdom project, so you don’t need to remote control a browser, but I’m not sure how well that will work on large projects.

Anyway - I’ll keep you updated.

Visa is down

2010-12-09T00:00:00-08:00

Visa.com and mastercard.com are down (as of 3pm Thursday in New Zealand) from the anonymous ddos. For my part, I made a protest sign that shows my feeling about how Sweden is acting as an independent and sovereign nation with respect to Julian Assange:

Sweden, Americas Bitch

Social network over XMPP

2010-12-08T00:00:00-08:00

I thought I’d do a post showing how a social network over XMPP works. This isn’t any invention of mine, I’ll just point to the relevant specs and give my understanding of how it can work. I’m not an XMPP expert though, so take this with a pinch of salt.

User IDS

Users are identified by their jabber id, which looks like an email address - eg: bnolan@gmail.com or ben@diaspora-x.com. Users have to be associated with a jabber server, but they can use any client they want. For example I could use diaspora-x.com to access my bnolan@gmail.com jabber account.

nb: Diaspora-x won’t support gmail as a jabber server until gtalk supports oauth, since otherwise a diaspora-x seed would have to store google account passwords.

Pubsub

XMPP supports publish/subscribe via XEP-0060. This means that you can send a message to your xmpp server, with no to address specified. Everyone on your roster who is subscribing for those kind of events will get your message. If someone on your roster is offline, their xmpp server will store your message until they come back online.

Personal eventing protocol

XEP-0163 describes how any jabber user can act as a virtual pubsub service, and send their own status / events.

<iq from='juliet@capulet.lit/balcony' type='set' id='pub1'>
  <pubsub xmlns='http://jabber.org/protocol/pubsub'>
    <publish node='http://jabber.org/protocol/tune'>
      <item>
        <tune xmlns='http://jabber.org/protocol/tune'>
          <artist>Gerald Finzi</artist>
          <length>255</length>
          <source>Music for "Love's Labors Lost" (Suite for small orchestra)</source>
          <title>Introduction (Allegro vigoroso)</title>
          <track>1</track>
        </tune>
      </item>
    </publish>
  </pubsub>
</iq>

Sending a song update using the tunes extension

This pubsub activity is commonly used by user tunes so you can see what your friends are listening to. If you’ve ever seen someone updating their adium status with song names from itunes, they might’ve been using XEP-0118.

Another use of publish / subscribe is microblogging - described in XEP-0277.

Microblogging

Microblogging over xmpp create a node urn:xmpp:microblog:0 on each user, and then provides a protocol for sending atom messages over xmpp. This protocol has been expanded upon by the Vodafone research group as Activity Streams over XMPP which lets you publish, update and delete posts. It also gives a vocabulary of verbs that you can use to describe your activity, so you can like, bookmark or group posts.

A client doesn’t have to implement all the verbs. Diaspora*x only implements the like and post protocols (replies to posts are posts as well), but as you fill out the functionality, you have a predefined vocabulary so that clients can work together.

Alternate clients

I have a few goals for diaspora*x, if I keep working on it.

Get federation working reliably with other servers, currently the xmpp connector is incomplete, although it shouldn’t take long to get going. I’d also like to test the federation against another social network, maybe buddycloud, or onesocialweb.
Create a desktop client using strophe.js, that connects to the jabber server directly (via BOSH) and shows how a client can be implemented in the simplest possible way.

A part of my job I think, is to demistify social networking over xmpp, as it’s quite a simple idea and easy to implement on the current specs.

I also have to read up on ostatus and work out if and how diaspora*x should work with ostatus. If it makes sense to add ostatus support, that would mean anyone on the status.net network would be friendable via a disapora*x seed.

Diaspora*x

2010-12-07T00:00:00-08:00

I like Facebook, and at the moment I think it does a great job of managing privacy on the Facebook site. My only concern is that of censorship. If the US government asks Facebook to censor something, Facebook has to acquiesce to their request.

I don’t believe this is a problem in the short term (if the wikileaks facebook page goes offline no one is going to die), but I do think the idea of having a distributed social network - where there is no central point of control or central point of failure - is a cool idea.

So I was excited to learn about Diaspora. It’s clearly something people are interested in, their fundraising was 20x oversubscribed. So when they did a public release, I was keen to check out the code and try it out.

The current alpha release of Diaspora is interesting, but it clearly has a ways to go before it’s ready for day-to-day use. The biggest problem I found with it is that it doesn’t support XMPP for communication between different seeds. XMPP is also known as Jabber, and it’s the protocol that gtalk and ichat use to communicate. Even facebook uses xmpp in places to allow you to interact with facebook chat.

Adding XMPP support to Diaspora

So I grabbed a few bits out of the Diaspora trunk and built a little app to demonstrate how a basic distributed social network could work. You can try out my result of a few days coding here:

» diaspora-x.com

I’m not sure how much more work I’m going to do on this project, I’ve proven the idea works to myself, and I’d like Diaspora proper to start work on XMPP support for the main project.

However - if you want me to add a feature to the diaspora-x website, put your requests into the github issue tracker and I’ll see what I can do.

There are installation instructions here if you want to try running Diaspora*x yourself.

Offending people

2010-12-06T00:00:00-08:00

Any time you click a broken link (like this one for the october update) on the Diaspora site, you get this guy staring back at you.

Oops. My bad.

Except, well I didn’t do anything wrong, except I clicked a link that you broke on your site. And now I feel like a fool.

Cool graphic design. Really bad user experience.

On how things stay the same

2010-11-30T00:00:00-08:00

When you’re young and you have problems or obstacles, you are comforted by the fact that you’ll get through this and in the future you won’t have such problems.

It’s interesting how no water how as you get older and more experienced, you get much better at solving your problems (oh it’s a X so i’ll do a Y), but that feeling of ‘my god, what the fuck am i going to do’ doesn’t go away. It just is more brief before you accept ‘oh well, i’ll just have to use a self tapping bit and use an oversized thread.

Well, that’s how it’s been so far - be interesting to see what the next 20 years of problem solving will reveal.

Horizontal sharding on Postgres

2010-11-29T00:00:00-08:00

Working on twitterplaces I get around 2 million tweets per day. The importer is a streaming curl session that I pipe into a ruby script that analyzes the tweet and sticks it into a postgres database.

Tweet analysis

Some of the analysis I do for each tweet:

Is it an @ reply?
What place_id is being referred to?
Does the user seem spammy?
What neighbourhood does this tweet belong to?
What city does this tweet belong to?

This has worked really well and even though its low tech, when combined with a monit script, it’s easy to control and robust.

The problem

The problem is that after 20 days, you have more than 50 million tweets and postgres starts to grind to a halt when inserting data.

If you had forgotten all of your university level mathematics - you might model the relationship between time to insert and the number of tweets like so:

lim T(insert_tweet) as SUM(tweets) &rarr; 50m = &#8734;

The solution

The obvious solution is to delete the old tweets out of the database. Maybe a cron job like this:

delete from tweets where created_at > 7.days.ago

Except every time you do that (and the VACUUM that you have to run afterwards), postgres will stop responding for about an hour, giving a theoretical uptime limit of 0.95. Since I’m aiming for a uptime of at least 3 sixes (0.9666), I decided to look int other options.

NoSQL

The sexy thing to do would be to use mongodb, couchdb or cassandra. But that migration, and writing all the map-reduce functions, plus removing activerecord and rewriting all my views - would probably take more than the 60 minutes I assigned to this problem.

So… What can we do with postgres?

Daily sharding

The easiest solution is a variant on this, where you use table inheritance to insert each days tweets into a distinct table, and then query the parent table to get an aggregate view.

So you just have a daily cronjob to drop any tweet tables more than 7 days old, and then create a new table for tomorrows tweets. Since tables are stored seperately on disk, the drop table is essentially free.

Performance concerns

Indexes can’t be built up across multiple tables (and if you did do that, you’d need to rebuild them each day when you dropped 1/8th of the indexed data), so badly written queries could end up with a sequential scan across all tables. I’ve only had the new system running for a few hours, so I’ll know more about performance implications later this week, but since most of my queries run on only the last 24 hours, I hope I can hint postgres to query the most recent table (using the daily indexes) first and only then move back to the second most recent table.

The code

Nice and simple - a cronjob to create tomorrows table…

def self.create_tomorrows_table
  date = 1.days.from_now.to_date
  date_string = date.to_s.gsub(/-/,'_')

  sql =<< EOF
    CREATE TABLE tweets_#{date_string} ( 
      CHECK ( DATE(created_at) = DATE('#{date}') )
     ) INHERITS (tweets);
  EOF

  ActiveRecord::Base.connection.execute sql

end

And then a sharded save method (note that I have a very customized tweet import job, so I don’t have to worry about tweet ids / saving associations).

def sharded_save!
  date = self.created_at.to_date
  date_string = date.to_s.gsub(/-/,'_')

  quoted_attributes = attributes_with_quotes

  sql = << EOF
    INSERT INTO 
      tweets_#{date_string} (#{quoted_column_names.join(', ')})
    VALUES
      (#{quoted_attributes.values.join(', ')})
  EOF

  ActiveRecord::Base.connection.execute sql
end

To be continued…

So this is just day one, I’ll find out more about this scheme later in the week, but I (yet again) was impressed by how nice a modern postgresql installation is. Well done the postgres team.

Jquery mobile on Nokia

2010-11-25T00:00:00-08:00

My Nokia 5230 arrived today. What a cool phone for 120€! It has satellite navigation (with a suction cup to stick it on the windscreen), a big screen, decent browser and facebook right on the front page. It’s no iphone, but it’s 1/5th the price.

As well as using the phone to navigate our way to the some quiet beaches north of Brisbane, I’m using it to debug weheartplaces, so I can get my app on the Ovi store.

First attempt

I loaded up jquerymobile.com on the nokia and was sorely disapointed, the current alpha of Jquery mobile doesn’t support Nokia S60 v5 (s60 v5 is the touch-screen interface for Nokias) browsers. I quickly set about to see how much work was needed to get things going.

An hour later

The backbone mobile demo running on a 5230

It works! I created and solved two issues on the Jquery Mobile github - one about history and another about window.innerHeight - and suddenly it all started working.

There are a few things missing in the current S60 implementation, there is no inter-screen animation and some of the CSS looks a bit rough on the symbian browser, but it’s totally functional and looks nice enough.

Fixes needed to jquery mobile for s60

Promote s60 v5 to a grade a browser
Fix the history tracking (solution at issue 542)
Fix window.innerHeight in jquery (solution at issue 542 )
Fix inter page animations
Fix png / css ugliness

One code base, at least three platforms

The backbone mobile demo running on iPhone

It’s exciting to be able to write an application once, in a comfortable development environment (Safari on the desktop), then deploy the same code to iPhone, Android and Nokia. I expect start-ups will begin using this toolchain to build apps for multiple appstores at once. To try out the jquery mobile + backbone app on your Nokia (you’ll need a s60 v5 device), Android or iPhone, go to my backbone mobile demo.

Backbone Mobile Example

2010-11-24T00:00:00-08:00

After my post yesterday - I decided to extract out an example app from my current work.

A backbone mobile app running in Safari

You can try the app in your browser, or view the source code on github.

Technology

I’m using the Foursquare API, Backbone.js, Underscore.js, CoffeeScript, jQuery and jQuery Mobile. The application is written in CoffeeScript - if you make changes to application.coffee - you will need to rerun coffeescript to recompile application.js.

If you aren’t up to date with coffeescript you can probably read through application.js and understand most of the code. I also cached a foursquare venue API search result as foursquare.json. The foursquare API doesn’t support JSONP so I couldn’t use it directly.

Patch to jQuery Mobile

There is only patch require to the libraries - one that I discussed in my previous post to make jQuery route the URLs correctly.

Index.html

The index.html has very little in it, just the outline HTML for the first page to be displayed. The actual page content is generated by HomeView in application.coffee. If you haven’t seen the data-role stuff before, I recommend you read through the jquery mobile docs.

Application.coffee

Pretty much the entire application is written in application.coffee.

app

A namespace with a few helpers I use to work with jquery mobile better. activePage returns the currently displayed page. reapplyStyles make jQuery mobile convert elements tagged with data-role=”whatever” into correctly displayed jquery mobile widgets. The next version of jQuery mobile might fix the .page() method, and so reapplyStyles can be removed.

Venue class

class Venue extends Backbone.Model
  getName: ->
    @get('name')

  getAddress: ->
    [@get('address'), @get('city'), @get('state')].join ", "

  getImageUrl: ->
    @get('photo_url')

  getLatitude: ->
    @get('geolat')

  getLongitude: ->
    @get('geolong')

  getMapUrl: (width, height) ->
    width ||= 300
    height ||= 220

    "http://maps.google.com/maps/api/staticmap?center=#{@getLatitude()},#{@getLongitude()}&zoom=14&size=#{width}x#{height}&maptype=terrain&markers=color:red|#{@getLatitude()},#{@getLongitude()}&sensor=false"

This is a backbone model that takes a hash of data and makes it available with a few helper methods.

VenueCollection

class VenueCollection extends Backbone.Collection
  model : Venue

  constructor: ->
    @refresh($FOURSQUARE_JSON)

This is instantiated as Venues (try running Venues.models in your javascript console and you’ll see all the loaded venues). When the collection is created, I call refresh with the FOURSQUARE_JSON (which is loaded from foursquare.js) which populates the collection immediately.

EditVenueView

class EditVenueView extends Backbone.View
  constructor: ->
    super

    # Get the active page from jquery mobile. We need to keep track of what this
    # dom element is so that we can refresh the page when the page is no longer active.
    @el = app.activePage()

    @template = _.template('''
    <form action="#venue-<%= venue.cid %>-update" method="post">

      <div data-role="fieldcontain">
        <label>Name</label>
        <input type="text" value="<%= venue.getName() %>" name="name" />
      </div>
  
      <div data-role="fieldcontain">
        <label>Address</label>
        <input type="text" value="<%= venue.get('address') %>" name="address" />
      </div>
  
      <div data-role="fieldcontain">
        <label>City</label>
        <input type="text" value="<%= venue.get('city') %>" name="city" />
      </div>
  
      <div data-role="fieldcontain">
        <label>State</label>
        <input type="text" value="<%= venue.get('state') %>" name="state" />
      </div>
  
      <button type="submit" data-role="button">Save</button>
    </form>
    ''')

    # Watch for changes to the model and redraw the view
    @model.bind 'change', @render

    # Draw the view
    @render()

  events : {
    "submit form" : "onSubmit"
  }

  onSubmit: (e) ->
    @model.set {
      name : @$("input[name='name']").val(),
      address : @$("input[name='address']").val(),
      city : @$("input[name='city']").val(),
      state : @$("input[name='state']").val()
    }

    @model.trigger('change')

    app.goBack()

    e.preventDefault()
    e.stopPropagation()

  render: =>
    # Set the name of the page
    @el.find('h1').text("Editing #{@model.getName()}")

    # Render the content
    @el.find('.ui-content').html(@template({venue : @model}))

    # A hacky way of reapplying the jquery mobile styles
    app.reapplyStyles(@el)

    # Delegate from the events hash
    @delegateEvents()

The first view. It sets the active page to the @el attribute, so that it can refresh the page later on (remember that the activePage may change when the user navigates to another page, so we need to be able to refer to what element we drew the form in). Then we create a template using underscore.template. The underscore template library uses ERB style syntax to generate HTML. Remember that the ERBs are evaluated using the javascript interpreter, not coffeescript, so there’s a bit of mixing and matching of coding styles going on here.

We bind to the change event on the model so that if model is updated somewhere else, this form is automatically re rendered. We watch for the submit event on the form (by calling @delegateEvents), so that we can handle the form submission ourselves in onSubmit.

onSubmit sets some properties on the model by reading them out of the inputs, then triggers change on the model. This would normally be a call to @model.save(), but we don’t have a datastore to save the data too - so we just throw away the changes. In my own apps I use localStorage to persist the changes to the data, and then have a sync method that synchronises the local store with my production servers.

render sets the title of the page, renders the template into the .ui-content div, calls reapplyStyles so that jquery mobile can do it’s magic, and then calls delegateEvents so that the events listed in the @events hash is applied to all the created elements.

ShowVenueView

Simpler than the EditVenueView, except it generates some internal urls:

<a href="#venues-<%= venue.cid %>-edit">Edit</a>

This URL is recognized by backbone (see the routes in the controller) and will cause the edit method on the HomeController to be called. We also bind to change here as well, so that when you save changes to the form, it automatically redraws the form. This is one of the main advantages of backbone.js - you can loosely couple views and not have to think of what views need to be redrawn when you make a change to a model, it happens automatically.

HomeView

Displays the list of places. Uses the .each() method that underscore.js makes available (or delegates to the native browser implementation) to iterate over all the venues and create a link to them.

HomeController

class HomeController extends Backbone.Controller
  routes :
    "venues-:cid-edit" : "edit"
    "venues-:cid" : "show"
    "home"  : "home"

  constructor: ->
    super
    @_views = {}

  home : ->
    @_views['home'] ||= new HomeView

  show: (cid) ->
    @_views["venues-#{cid}"] ||= new ShowVenueView { model : Venues.getByCid(cid) }

  edit: (cid) ->
    @_views["venues-#{cid}-edit"] ||= new EditVenueView { model : Venues.getByCid(cid) }

I only have one controller in this app, but you could easily split it into two (a home controller and a venues controller). First up - we define the routes that we match. Note the routes have to be in an order of decreasing specificity, otherwise venues-:cid would gobble up venues-1235-edit.

I created @_views hash that I use to keep track of all the views that I have created, since I don’t want to generate two instances of the same view. This is quite a naive implementation, because views can be created dozens of times if the user clicked on every single Venue. We would either want to expire old views, or re-use views - but I haven’t worked out a way to manage that yet.

Home, show and edit are all called by the Backbone router (see Backbone.history) when the anchor part of the current url (event hashchange) changes. Each action is called with the parameters extracted from the routing hash. So all we have to do is instantiate the view and our app is ready to go.

Finally, when the document is ready we start the backbone router and render the initial view.

Conclusion

I’ve done less than a weeks development with jquery mobile (I’d done a lot more using jqTouch), but it seems like a nice complement to backbone.js. It’s fantastic that you can build an app using your current web experience, that can be sold in the Ovi Store (via Nokia Webruntime), Android Marketplace or iTunes Store (Phonegap).

I’ll try and keep things updated as I build out my own app, and see how things scale up.

Backbone.js and Jquery mobile

2010-11-23T00:00:00-08:00

The idea behind Weheartplaces is that I want to be able to bookmark places to go on my desktop computer, and then sync these places to my mobile so I can access them offline.

I’ve created a json API to weheartplaces, here are my bookmarks in json format. I’m loading these bookmarks using backbone.js and then displaying them using jquery mobile. Here’s a little breakdown of how I’m doing it:

List view

Create a list view like this:

<ul data-role="listview" data-inset="true" data-theme="c" data-dividertheme="b">
</ul>

This is automatically converted into a nice looking list view by jquery mobile. You then iterate over all the elements in your Backbone collection to add them to the list:

for bookmark in Bookmarks.models
  a = $("<a />")

  a.text(bookmark.get('place').name)
  a.attr 'href', "#bookmarks-#{bookmark.cid}"

  a.wrap("<li />").parent().appendTo(ul)

Notice the url that each bookmark points to - #bookmarks-12345. I use backbone routes to recognize this URL and display the correct content.

Combining the jquery and backbone routers

When you click a link - jquery mobile intercepts the click and looks for a div with the same name as the anchor fragment of the url you just clicked on. So for example:

<a href="#home">Home</a>

Will cause jquery to show this div:

<div id="home">some content...</div>

The problem is that in my list view, I’m linking to pages that don’t exist yet - so I have to add some code to the jquery mobile routing code to create and show an empty div.

// This code goes in $.mobile.changePage...

if( url ){
	to = $( "[id='" + url + "']" ),
	fileUrl = getFileURL(url);

  if(to.length == 0){ // Page could not be found
    console.log("Routing to " + url + "...");
    
    to = $("<div data-role='page' id='" + url + "'><div data-role='header'><h1>&nbsp;</h1></div><div data-role='content'><img src='images/ajax-loader.png' /></div></div>").appendTo('body')
    
    enhancePage();
  }
}

Jquery will display this page, and change the anchor fragment of the current URL. Backbone detects the URL change by watching for a hashchange event, and will then call your matching route - in my case:

class HomeController extends Backbone.Controller
  routes :
    "home"  : "home"
    "bookmarks-:cid" : "show"

Showing the bookmark

Once that magic has happened, the show method on the HomeController will be called:

show: (cid) ->
  new ShowBookmarkView { model : Bookmarks.getByCid(cid) }

This creates a new ShowBookmarkView, and that view generates the HTML (using underscore.js templates) that is inserted into the currently active page.

$(".ui-page-active").html(@template({bookmark : @model}))

If I get time I will put a small demo file up on github to show how all this hangs together - but feel free to post questions below.

Back from Hiatus

2010-11-22T00:00:00-08:00

Sorry I’ve neglected my blog for two weeks. I was finishing up a contract, and then took a week break to work on some personal projects, as well as sort out various life things.

Travelling around New Zealand

Cathedral Cove

In February, my girlfriend and I are travelling around New Zealand for 2 months, before heading off to Brisbane, then various other destinations around the world. To get the most out of our trip, we’ve been planning a bunch of places to go on weheartplaces. It’s still early days - but if you go to my guides page you can see pictures and addresses for some of the campgrounds, breweries and beaches we want to visit while we’re travelling around.

The DOC (Department of Conservation) campsites are the best, since they cost only $6-$10 a night, and tend to be in more remote areas.

I had an interesting talk with some New Zealand based travel guys about making their campsite data accessible offline, so if that comes up, Rissa and I should be able to travel around with a full database of every nice freedom or DOC campsite available.

My favourite places in the world.

I made a quick list of my Favourite places in the world:

Midnight Espresso, Hofbraükeller, Cathedral Cove, and Ferchensee.

Nokia 5230

I bought another new Nokia, this time a Nokia 5230, which has the more modern Web Runtime 7.2, so that I can build some backbone.js applications that run on IOS, Android or Nokia.

Compared to some of my other projects, which are pure ruby on rails, no javascript at all - my Phonegap / Web Runtime apps have a gnarly technology stack:

Coffeescript
Backbone.js
Jquery mobile
Less CSS
Qunit
Phonegap

I’ll talk more about phonegap and backbone.js + jquery mobile later this week.

Level Interaction

2010-11-04T00:00:00-07:00

I’ve just posted a new video of Hutch. New features include terrain generation, better collision detection, item interaction, tile interaction. I’ve been looking around the web and also found a few people doing truly amazing javascript stuff:

It really looks like Javascript gaming is about to take off!

Game design

2010-11-03T00:00:00-07:00

I’ve been playing around with Hutch lately, and have been reading some game design articles. I found this pearl of wisdom from the developer of some old Commodore 64 games that I used to love as a kid.

(As an aside, said C64 developer now writes insurance software because it pays better).

The most interesting part of the article I thought was the developers game design process:

We investigated the platform and designed and wrote graphics architecture to exploit some feature of the machine.
We considered game environments that utilised that feature.
We create a player in a prototype level and worked on the control and movement.
We decided on the game idea and added enemies to the test level.
We worked on real levels constantly coming up with new ideas as we progressed.
We reviewed the levels , adding new ideas in where necessary

This is very similar to how I’ve been working on Hutch. Invent the graphics engine, think up game environments you can render in realtime, add player movement and control. The next stage is to add items and enemies.

Hopefully I’ll get time to do that later in the week.

Twitterplaces

2010-10-28T00:00:00-07:00

I went to a R users group in melbourne and got all turned on by data analysis, so I went home and started crawling all the tweets in the world and sticking them into a Postgres database. I then went and build this site.

I keep playing around with the site and adding random features and new pages, but I’d like some feedback on what I could do so that people actually found the site useful. It’s cool building a site for myself, but it’d be cool if more than 10 people a day used the site. What do you lot think would be some concrete features to add to the site?

Here’s some of the current pages that I think are interesting but could do with some work:

http://www.twitterplaces.com/muc
http://www.twitterplaces.com/users/dens
http://www.twitterplaces.com/sfo/twitter-hq
http://www.twitterplaces.com/sfo/mission

Some technical infos:

I crawl the data using the streaming API, all geotagged tweets in the world (about 2m/day). The crawler is written in ruby and does a bunch of caching so it doesn’t have to hit up postgres too much.

I use postgis so I can lookup bounding neighbourhoods and cities for points based on lat / long.

Some of the queries are insanely slow (eg which users travel the world the most), so I need to work out a better way of generating results.

It’s running on a AMD X4 box with 4gb of ram, so seems to keep chugging along.

I use varnish on the front of the site to cache html for a few minutes after each visit.

TLDR: Have site, what features should I add?

Additions to Backbone views

2010-10-27T00:00:00-07:00

I took Matt McCrays excellent Coffeescript glue for Backbone.js and moved my saveChanges function into it.

The function serializes all the form elements in the rendered elements and updates the associated model. You can use this code by binding to ‘submit form’ using @handleEvents and calling saveChanges and suppressing the default form submission.

class View
  constructor: ->
    Backbone.View.apply(this, arguments)

  saveChanges: ->
    properties = {}

    for input in @el.find('input,textarea')
      value = $(input).val()
      properties[input.name] = value

    @model.set properties
    @model.save()

    app.saving()

_.extend(View::, Backbone.View.prototype)

A small example from Weheartplaces…

class TripInspector extends App.View
  constructor: ->
    super
    @template= JST["trip_inspector/show"] 
    @render() if @model?

  render: ->
    @el.html(@template { trip : @model })

    @handleEvents {
      'submit form' : 'submit'
    }

  submit: (e) =>
    e.preventDefault()
    @saveChanges()

this.TripInspector = TripInspector

Tagging

2010-10-26T00:00:00-07:00

I’ve been doing some work on getting the private beta of Weheartplaces out. I’m using tagging as a way to categorize places, so you can quickly add a place and tag as ‘todo’, ‘cafe’ or possibly ‘avoid’.

Places tagging interface for Weheartplaces

I found tagbox by Rafudu still works okay with current jquery. It’s a bit creaky, but with some CSS tweaks it worked okay out of the box.

Gisborne

2010-10-22T00:00:00-07:00

I’ve been away from the internet for a bit - chewing through some work for the good folks at Youdo (you should try their ITIL tool - Beetil) and doing various business development work. One of the most enjoyable business trips I’ve ever been on can be summarised by this photo:

A remote beach on the east cape of NZ

Four for four

2010-10-15T00:00:00-07:00

I just sat down at my Mac. Here’s a recap of the last hour.

Thirty minute bike ride on muddy singletrain
It stopped raining and the sun came out
As I got out of the shower my girlfriend rang up to ask what I wanted her to bring for lunch
I pressed play on iTunes and got some pimping Murs

Yeeeeeeeeeeah.

A modern webapp

2010-10-14T00:00:00-07:00

Want to write a modern webapp using lots of javascript? Here’s my recommended toolkit:

CoffeeScript for brevity
Jquery for sanity
Jquery UI (with Aristo) for your design and widgets
Underscore.js for your views (see _.template)
Backbone.js for your models andd controllerts
Jammit for asset packaging

I’ve only recently discovered backbone and jammit, but I have a lot of respect for Jeremy so they’re my bet.

Splatter

2010-10-12T00:00:00-07:00

After watching Exit Through The Gift Shop last night, I wanted to do something creative, so I played around with isometric projection using css3 and applying stencils to canvas. I came up with something vaguely similair to what I was imagining….

Designing a little world

My jqtouch fork

2010-10-08T00:00:00-07:00

Sorry for not blogging this week. I’ve been in a client office and furiously working on getting a sprint out. On Monday I had my product development day, so I started work on weheartplaces. The first iteration of WHP is due out by the end of October, and I plan to have it working on Iphone, Nokia and Android. To do this I’m using jqtouch which hasn’t seen a lot of work in the past year.

I forked jqtouch onto my github and have been pushing a few changes. They aren’t all ready for the primetime yet - but the main design changes I’ll be making:

URL routing scheme, so you can have controllers and actions to handle requests
Use pushState to have proper urls (instead of #location urls)
Specced and tested using Jasmine
Better form support

I also have some ideas about how I can make an analog to jqtouch that will work with the Web Runtime that Nokia supports for development on their phones. I’m doing my jQTouch modifications in pure javascript, but the Weheartplaces application is written using Coffeescript.

Silverstripe hosting

2010-10-05T00:00:00-07:00

I have a little joke going that I think a hosted silverstripe would be quite a good competitor to wordpress.

The silverstripe admin interface with a few tweaks

Silverstripe is an excellent CMS built by some of my friends in Wellington. They have a big company now, over 30 people, lots of big clients - and they even have a product - Dawn. Wordpress.com is the free hosting provider that runs Wordpress MU and has approximately a bajillion blogs on it.

The revenue stream is that on wp.com, you have to pay to get premium features, like CSS editing, premium themes, custom domain names. I think it’s an awesome business model that the Silverstripe team should take a look at.

A preview of new design

2010-10-01T00:00:00-07:00

I have a new menu and header design coming for twitterplaces. A preview:

Random Walk

2010-09-30T00:00:00-07:00

For twitterplaces, I’ve been using the Random Walk theory of software development. I pick up the software each day, and work on whatever feature I feel like. There’s no plan, no vision, no long term overarching story that directs my work.

I just go to a page that looks scruffier than the rest, then tidy it up. Or play with the data and find another interesting report, then put that in the site somewhere. Occasionally I may have to do a bit of architecture or scaling work, but i don’t plan for it - I just do it when required. And I’m always pushing to make the code easily understandable. A random walk doesn’t mean spaghetti code.

It just means, don’t beat on feature a until it’s done. Work on a,b,c and invent feature d. That way you won’t wear out and you won’t become uninspired.

Sure this won’t work for your on-spec, on deadline project, but for your own projects, or experimentation in user interface design, just random walk through your project.

You may be surprised where you end up.

A graphviz plot of adjacent commits on twitterplaces

Against Me!

2010-09-29T00:00:00-07:00

My favourite band in the world cancelled their Australian and New Zealand tour.

Sad face.

The Art of Happiness

2010-09-28T00:00:00-07:00

I think this sums it up for me:

When you feel lacking in happiness. Don't delay feeling happy for when some future achievement has happened. Let yourself feel happy now.

Delayed gratification doesn’t apply to happiness.

An island of Javascript

2010-09-27T00:00:00-07:00

I ready about the sale of Dextrose to Zynga by Paul Baukaus, and it got me motivated to poke around with some html5 gaming again (a nice sunday morning thing to do). The outcome was an island generator using three.js and jquery ui.

Rudimentary but functional. 3d javascript for the future!

When you start the demo, you just get an empty sea, but move the cube around with the arrow keys and hold down space to raise some mountains out of the sea. The demo has a bad case of z-fighting, but I only spent an hour or so on this, so I’ll look into those issues in the future sometime.

Three.js

Three is a great library from one of the authors of papervision 3d (so they know what they’re doing when it comes to browser-based graphics). It is surprisingly fast on modern browsers - the slowest thing seems to be the canvas drawing calls, which aren’t hardware accelerated. The great thing about doing javascript work though, is that browsers are being updated so often at the moment (Safari, IE9, Firefox and Chrome) that what is slow but workable today, will be liquid smooth in 6 months time (assuming browser development keeps going like this).

Three also has output support for canvas and webgl. I’m using canvas in this demo.

A 3d game engine

I’d like to play with this demo some more, and turn it into a mini game. I’m currently thinking of using three to render to multiple canvases, and position the camera so that the terrain doesn’t obscure itself, and I can use the browsers native z-order stacking for elements on top of the map. This should make a realtime 3d world doable without pegging the CPU (I have a strong aversion to pinning the CPU to draw a 400 face terrain when this macbook can play TF2 at native resolution).

Then add multiplayer via websockets and node.js and you’d have that multiplayer canvas game that Simon Willison predicted.

New Twitter Analyzed

2010-09-24T00:00:00-07:00

I was interested to read the twitter engineering teams overview of how the newtwitter works.

This is the kind of app that is becoming more and more common, as it becomes viable to build apps in pure javascript, with just a json API as the backend. There are downsides to this kind of development, mostly in the inability for google to index the site - but for an app like twitter which are indexed by google using the pubsub interface (afaik), there’s no reason not to provide a pure javascript / html5 interface for modern browsers.

I haven’t got new twitter yet - so I hit up a friend and got him to send me the URLs for the new twitter javascript. I guessed the pre-closure URLS, tidied them up using html tidy and textmate - and we have these:

Base.js

Base.js seems to contain their extensions to jquery to create the twitter ‘framework’, eg:

Logging
Debugging
Stack traces
Browser detection
Analytics
Page state / history
Load templates
Form validation
Help
Autocomplete

It also has some code that uses the anywhere api (I think), to find a user by screenname. It has a bunch of code that has either been handrolled, or gathered together from different jquery plugins (eg timeAgo, numberWithDelimiter).

The really interesting stuff is in phoenix.js though.

Phoenix.js

Newtwitter uses Mustache for their templates. Many of these templates are included in phoenix.js, as twttr.templates. The templates are internationalized and use classes liberally - eg stream-tab-following, stream-tab-pending, delete-saved-search.

If you search on any of these classnames - you’ll find code later in phoenix.js that attaches the appropriate event handler to the html generated by mustache. For example - this fragment:

<div class='delete-saved-search'>
  \n <a href='#' class='delete-saved-search-x'></a>\n <a href='#'>Remove saved search</a>\n
</div>",

Has behaviour attached to it like so:

click({
".delete-saved-search": function(B, A) {
    twttr.dialogs.deleteSavedSearch(twttr.bind(this,
    // ....
    B.stopPropagation();
    B.preventDefault();
    return false
})

I’ve munged the code up a bit here, but you get the basic idea. Phoenix.js includes jquery (the latest release, 1.4.2), Jquery UI (1.8.4), Modernizer (a library for detecting browser support for various html5 functionality), Linkify.

This was just a quick analysis of the newtwitter codebase - but I found all the mentions of twttr.view to be interesting - it seems like the twitter engineering team have built quite a strong framework for javascript apps - I wonder if they’ll release their MVC framework for other developers to create similair apps?

Varnish and Rails

2010-09-23T00:00:00-07:00

I got my R scripts running on production for twitterplaces - so if you browse to Wellington or London, you can see twitter densities. That’s all well and good and very rough, there’s a lot of work to do:

The time slider looks rubbish
The tweet frequency histogram isn’t adjusted for the cities time offset
The extents of the different cities are different, so you get elongated contours
The contour needs to be subdivided for smoothness
It would be good to weigh against multiple tweets from a user

But the first problem, is that the contours are exceptionally expensive to convert to json, which is done on every page load by ruby. I wanted to have an easy way to cache a page for 60 seconds, so that if the page gets ‘seagulled’ (a thousand people follow a link from techcrunch, look at the page for a second, then go back to techcrunch to comment ‘it doesnt scale get nosql why not use php’) the site doesn’t go down.

Ask Koz

I originally was going to do some apache rewriterule hax, but first I asked @nzkoz, who told me to look into Varnish. Varnish looked good so I gave it a go.

Varnish

I followed the installation instructions for Ubuntu, then edited /etc/defaults/varnish to make varnish listen on port 80.

DAEMON_OPTS="-a :80\
             -T localhost:6082 \

I edited /etc/apache2/ports.conf to move apache over to port 8080:

NameVirtualHost *:8080
Listen 8080

And edited the appropriate sites-available scripts to move the virtual hosts over to the new port. Then I edited my /etc/varnish/default.vcl (make sure you install Varnish 2.x, not the 1.x that comes with some old varieties of Ubuntu) like so:

sub vcl_recv {
        unset req.http.Cookie;
}
  
sub vcl_fetch {
        if (req.request == "GET") {
                unset beresp.http.Set-Cookie;
                set beresp.cacheable = true;
                set beresp.ttl = 60s;
        }

        if (req.url ~ "^/images/" || req.url ~ "^/javascripts" || req.url ~ "^/stylesheets"){
                set beresp.ttl = 15m;
        }
}

All GETs are cached for 60 seconds. Javascripts, images and stylesheets are cached (on the client side) for 15 minutes.

Note that this varnish config will annihilate any site that uses cookies - but twitterplaces (and weheartplaces) currently don’t use cookies at all - so everyones looks at the same pages. I then restarted varnish and voila, we have a fast cached site that can cope with some seagulling.

To check that things are working - I tailed my rails log, hit the site up with curl -I http://twitterplaces.com/wlg/ and each request was served correctly, but didn’t show up in the rails logs until 60 seconds had elapsed. Varnish is very clever and will serve the old version of the cached page until the new version has finished regenerating. Varnish really does seem like a neat bit of kit.

Elapsed time, from learning of Varnish, to installing, configuring, testing and writing this blog post - 45 minutes.

Moving to production

2010-09-22T00:00:00-07:00

I lied. I wasn’t really building twitterplaces, I just got back from 5 days snowboarding, relaxing and reading books by the fire. Now i’m back in the office and started the week with getting my production box set up so I can deploy the next release of twitterplaces.

Monit

I set up a monit task to run my tweet importer and restart the process if it dies. The monit task looks like:

check process importer
	with pidfile /tmp/importer.pid
    	start program = "/usr/bin/sudo -u ben /var/www/tp/script/runner -e production Tweet.import" with timeout 20 seconds
    	stop program  = "/bin/kill -TERM `cat /tmp/importer.pid`"

Note that this job requires monit 5 since it uses the with timeout parameter. I also added this line to my importer script so that it spits out the process id.

`echo #{Process.pid} > /tmp/importer.pid`

R and ODBC

For my R scripts, I’ve been using the postgres adapter so that I can query directly from within R. It turns out the package I was using on the desktop wasn’t available on ubuntu, so I followed the instructions here and here to get RODBC set up with postgres.

Future tasks

I have a few things left to do before the new release is complete and ready for people to use:

Data massaging - the production database needs massaging and normalization of the bounding boxes for cities - I’ve been doing this adhoc, but it needs to be integrated into the twitter import process.
Data analysis cronjobs - some of the data analysis jobs are best run hourly (for example the contour map generation), so these need to be scheduled as a cronjob
Postgres partitioning - twitterplaces now imports over 2m tweets a day. To reduce the load on postgres, I’m going to use partitioning to reduce the size of the indices and quicker data truncation (I can just drop week old tables, instead of a delete and vacuum).

Hopefully i’ll be able to get some of that out later this week.

A break in transmission

2010-09-16T00:00:00-07:00

I’m taking a week off blogging so I can focus on building. Please excuse my silence - and I look forward to resuming communications with you all sometime next week.

Manners mall redevelopment

Twitterplaces Next Generation

2010-09-15T00:00:00-07:00

Ever since I discovered that you can index all of the worlds geotagged tweets with Twitter, I’ve been working on the next generation of twitterplaces. It’s a challenge. The app has to work on datasets the size of Wellington (around 110 tweets per day) to New York (around 15k tweets per day) and provide an intimate experience for every user.

One of the biggest problems with displaying lots of data on a map is that you end up with massive clouds of google markers all over the map. These look okay when there’s less than 5 markers, but above that, clicking on many markers is a bad user experience, it’s hard to get an aggregate view of what’s going on.

Viewing wellington tweets grouped by contour density maps

I’ve already posted about using R to create contour maps of tweet density. I’ve now taken that code and changed it from a manual job, into something automated. It’s not production ready yet - but the idea is:

Contour generator (R script) runs hourly on top 1000 cities
R queries postgres for tweets in the last hour
Generate the density matrix, and generate contour lines
Convert contours into WKT polygons and save to the database

When you display a city page - you export the contour polygons into a json array (indexed by hour) and then give the user some UI for changing what time window they are looking at.

Realtime contour generation

I’m not in a hurry to try and do this in realtime, but it has been at the back of my mind. My bookmarks tagged contour have my notes on the basic algorithm. Any query that filters keyword, user_id, generate contour maps of an individual suburb, do language analysis, sentiment analysis, etc - would generate data suitable for converting into contour maps.

If I wanted to be able to generate this data in realtime, there are two ways I can see of doing it.

Farm the raw tweet data off to the browser and generate the contour lines in Javascript
An optimized C++ app that uses sse / simd to generate the contour lines in a fraction of the time of R

Or - I could charge customers monthly, fire up another ec2 instance and keep using the existing queries in R.

Panasonic Lumix and the Optimal Prime

2010-09-14T00:00:00-07:00

I’ve been waiting to make this post for a few weeks now. I bought a Pansonic Lumix GF1 with the 20mm prime (non-zoomable) lens. It’s a real beauty! It takes lovely photos, is a joy to use (nice and fast) and fits in my coat pocket. I bought it as a present to myself for three months working for Lonely Planet in Melbourne.

Choosing the GF1

I got interested in a camera with a nice prime lens after seeing all Maddis epic photos. He uses a sony Alpha and the minolta beercan lens. I started researching and quickly became interested in getting an a330 and hunting down some 80s lenses - but as I read around the net about small form factor dslrs, I kept seeing micro four thirds being mentioned, in particular the GF1.

This review at dpreview and craigmods gf1 review were the articles that really sold me on the GF1. It’ll be fun to pack up our house into a big box and then go travel the world with this camera hanging off my neck.

Sweet Mothers Kitchen. Taken with the 20mm pancake lens

See more photos from last night on this flickr set.

A world of tweets

2010-09-13T00:00:00-07:00

The Twitter streaming API now lets you recieve all geotagged tweets, not just geotagged tweets in specified areas. This means that with a command like:

curl -d "locations=-180,-90,180,90" http://stream.twitter.com/1/statuses/filter.json -uuser:pass

You can recieve all the geotagged tweets twitter has. I updated my crawler to run against this complete dataset, fixed a few lingering bugs, and then let my crawler run for a few hours:

o's indicate exact location, .'s indicate neighbourhood

I then exported my tweets, and plotted x vs y in R:

plot(tweets$x, tweets$y); world(add=TRUE)

You can clearly see a world map forming, with a great concentration of geotagged tweets in north america, europe, japan and indonesia, as well as the outlines of Australia and New Zealand.

What if we cluster the points using kmeans?

Clustered into 5 groups using kmeans squared

We get 5 clusters, western us, eastern us, europe, asia and south america. Looks like Texas is an eastern state then. It’s kind of scary that Africa is so sparse that it gets gobbled up by Europe, South America and Asia.

I’m keen to see what a histogram of seconds from midnight (what I call epoch) and longitude will look like - but i’ll have to wait til I have 24 hours of data. It looks like I’ll end up with about 2 million tweets per 24 hour period, which should make a good dataset.

Contour maps of tweet density

2010-09-10T00:00:00-07:00

I regenerated by tweet density graphs using contour lines, and got this nice looking animation of tweet density in london over the course of a day.

24 hours of tweets in London

The important improvements:

Use a sliding time window

Before I was using hourly updates, which meant that the updates were quite abrupt, and I could have a maximum of 24 frames per movie. I used this subquery to pull out the number of seconds since midnight, so that I could use a sliding time window of one hour in R:

extract(epoch from created_at::time)

Use contour lines

The contour function in R generates really nice contours from heightmaps. They look nice, and are easily georeferenced.

Resample the density matrix

I sample the tweets in a 50x50 matrix. This works, but gives quite sharp contour lines. The image.smooth function (from the image library) came in handy here, as it uses a gaussian smoothing kernel on each element in the matrix. The default settings gives a nice smooth heightfield which can be plotted by contour

Next steps

After this - I worked out that I could call contourLines to get the line coordinates. Then by converting this to json using rjson, I plotted the contourlines onto a google map using polygons. Finally, I made it so that you can click on the polygons, and see what tweets and neighbourhoods are causing weird spikes of activity.

I’m busy with client work at the moment, so hopefully sometime next week I can push an update to twitterplaces that features these contour maps.

Imperial March

2010-09-08T00:00:00-07:00

One of the benefits from working at home is that I get to play trumpet and guitar as much as I want. Which means lots of time to practise.

For some reason, practising lots of the trumpet isn’t helping me get better at all.

Tweet map in R

2010-09-07T00:00:00-07:00

Just as a quick follow up to yesterdays post - this is the very rough visualization I came up with using R, openGL and ffmpeg.

The r script used was something like this. The texture isn’t correctly georeferenced, and needs to be flipped 180°, but this was as far as I got before having to move onto other work.

ourplot <- function (h) {
  north = 51.6
  south = 51.4
  east = 0.05
  west = -0.25

  df <- subset(tweets, x > west & x < east & y < north & y > south & hour == h)

  df <- rbind(df, data.frame(x=east, y=north, hour=-1))
  df <- rbind(df, data.frame(x=west, y=south, hour=-1))

  hdf <- hist2d(df$x, df$y, nbins=50, show=FALSE)

  persp3d( 
    hdf$x, 
    hdf$y, 
    hdf$counts, 
    ticktype="detailed",
    xlab="Longitude",
    ylab="Latitude",
    zlab="# Tweets",
    zlim=c(-30,50),
    expand=0.5, 
    shade=1.0, 
    specular="white",
    col="white",
    texture="/tmp/london.png",
    box=FALSE,
    axes=FALSE
  )

  rgl.snapshot(paste("/tmp/london-3d-",h,".png",sep=""))
}

for (hour in seq(0,24,by=1)) ourplot(hour)

I have a vague plan of taking the histogram matrix, exporting it to json and using protovis to redraw the graphs, possibly as a streamgraph.

Plotting Choropleths with r

2010-09-06T00:00:00-07:00

Choropleths are those heatmap style maps you often see. I’ve been keen to render some heatmaps out of the twitterplaces data I have, so I spent today learning R and using it to generate some graphs.

First up - I installed R using the os x binaries, then went right on to install rgdal.

Installing rgdal on snow leopard

To load map data, you first need to install the PROJ and GDAL frameworks. Then you need to install the R bindings for GDAL. When you try and install RGDAL from source - you may get this error:

Error: gdal-config not found
The gdal-config script distributed with GDAL could not be found.

To let R know where gdal and proj are - try this:

install.packages("rgdal", configure.args = " \
  --with-gdal-config=/Library/Frameworks/GDAL.framework/unix/bin/gdal-config \
  --with-proj-include=/Library/Frameworks/PROJ.framework/unix/include \
  --with-proj-lib=/Library/Frameworks/PROJ.framework/unix/lib \
",type="source")

R packages install super fast for me, rgdal took about 30 seconds to build and install. I’m beginning to see why people like this package.

Plotting california state boundies

Download the california shape files from the US census, then load the data and plot it:

cali=readOGR("/tmp","co06_d00")
plot(cali)

You should get a graph of californian country boundaries:

Plot of US census geo data

Now to extract data from my twitter data.

Generating neighbourhood shapefiles

As I import tweets from the twitter firehose, I create records in my place table that store the geometry data in a postgis polygon field. This means I can use pgsql2shp to extract the geometry data and create shape files that can be loaded into arcgis explorer, or imported into R using the readOGR command.

To generate my shape file - I run something like:

pgsql2shp -f london -u postgres tp_dev "select \
  name, geom \
from \
  places \
where \
  kind='neighborhood' and city_id=( \
    select id from cities where name='London' \
  )"

Which gives me 115 rows, all of the neighbourhoods in London that I have crawled so far. Plotting this data in R:

Plot of London twitter neighbourhoods

Twitter uses third party data to geocode your tweets. Sadly, the neighbourhood data isn’t free, so twitter can only share bounding boxes, which is why the map is all boxes.

In the future I might try querying openstreetmap for boundary data of administration areas 9 and 10, but I’m not sure how complete that data is.

Getting Tweet frequency

I re-ran the neighbourhood shape file dump, to include the number of users tweeting from each neighbourhood in london. The query ended up looking something like this:

select \
  id, geom, name, ( \
    select count(distinct user_id) from tweets where place_id=places.id \
  ) as user_count \
from \
  places \
where \
  city_id=( \
    select id from cities where name='London' \
  ) AND kind = 'neighborhood' \
order by \
  user_count desc;

I can then load this data into R, create a blue palette and plot it in 3 lines:

pal <- brewer.pal(9,"Blues")
plot(lon,col=pal[9 - (lon$USER_COUNT * 9 / max(lon$USER_COUNT))],border="white")

The graph however - is kinda disappointing, since the overlapping bounding boxes - combined with some very large neighbourhood areas, makes the graph pretty much useless.

Darker blue represents more tweets

Two dimensional histogram

After failing at my first graphing experiment, I tried plotting a two dimensional histogram of tweets. First up - spit out all the lats and longs of tweets from postgres. I haven’t installed the r<->postgres bindings yet, so I have to export to CSV and then import to R.

Enable CSV export from psql.

\f ',' 
\a
\pset footer off
\o /tmp/tweets.csv

Then query…

select 
  st_x(geom) as x, st_y(geom) as y, extract(hour from created_at) as hour 
from
  tweets 
where 
  geom is not null
and
  st_x(geom) < -100

This also gives us the hour of the day that the tweet was made, which might come in handy later. I loaded this data into R, then I could plot a scatter graph:

tweets <- read.csv("/tmp/tweets.csv")
plot(tweets$x, tweets$y)

Gives a random looking graph with a concentration of points around the mission.

A scattergraph of 1600 geotagged tweets around San Francisco

Okay - what does this look like as a 2d histogram?

hist2d(tweets$x,tweets$y)

A 2d histogram of tweets around San Francisco

Starting to get interesting - but still way too sparse. Let’s trim the area down to only include San Francisco proper, from the Marina District in the north, to the mission in the south.

city <- subset(tweets, y < 37.808 & y > 37.750 & x < -122.370 & x > -122.457)

Then let’s plot it in perspective 3d

h <- hist2d(city$x, city$y, nbins=25)
persp( 
  h$x, 
  h$y, 
  h$counts, 
  ticktype="detailed", 
  theta=30, 
  phi=30, 
  expand=0.5, 
  shade=0.5, 
  col="cyan", 
  ltheta=-30
)

User count in San Francisco rendered in a perspective view

The mission / cbd is in the top right of this graph, so you can see where most people are during the day.

Animating the graph over time

This is rendering 5 hours of data on one graph, but I was interested to see how the activity looked on an hourly basis, so I wrote a little R script to re-render the graph for each hour of the day, which may be a topic for tomorrows blog post.

nb: Note that I never actually created realistic looking choropleths in this post. ;D

Specialization and Art

2010-09-03T00:00:00-07:00

I like Heinleins quote, so instead of Follow Friday, I decided to try and post Art on Friday. So, an old charcoal drawing that’s been hanging in my office, and 90 seconds of my acoustic guitar.

Drawn at Vincents life drawing class in 2008

A warning - I made the guitar track up as I went along, so it’s pretty rough eh.

Something around Am and C, on my Tanglewood acoustic

The track was recorded by the imac’s internal microphone (I need to get a 9 volt battery so I can use the pick-ups on the acoustic), so sounds a bit like it was recorded in a tin can.

What to drink on Thursdays?

2010-09-02T00:00:00-07:00

I’ve been making a real effort to avoid coffee in the afternoon. I’ve made the discovery that although the first cup of coffee in the morning is awesome, afternoon coffees are just that ‘cup of stress’ that my vegan friends talk about.

So I made myself a handy guide to what I should drink, hour by hour, on Thurdays.

Tea... Wins!

The beer is optional, although I was planning on catching up with Herr Schommer after work.

Polygons and postgis

2010-09-01T00:00:00-07:00

I use postgis extensively in my geo-local development. If you’re working on a project that involves geo data and you haven’t seriously looked into postgis, you’re doing yourself a disfavour.

Postgres is really quite epic.

Postgis
R for Postgres
Python for Postgres
A nice stable API for creating C++ extensions

We used the C++ API to make a xapian plugin for zoomin, back in the day before tsearch2 became integrated into Postgresql.

Twitterplaces

Sam asked that I add polygons onto my twitterplaces maps, so that he could see where the boundaries of a neighbourhood are.

South of Mission on Twitterplaces

The twitter places API gives you the bounding boxes of a place, so I already had the information, it was just a case of how to store the bounding boxes. I decided to do it properly and use spatial_adapter and a :polygon column in my places table.

This meant I could use a method like so:

def bounding_box=(params)
  points = []

  params['coordinates'].first.each do |c|
    points.push GeoRuby::SimpleFeatures::Point.from_x_y(c.first, c.last)
  end

  self.latitude = points.collect(&:y).sum / points.length
  self.longitude = points.collect(&:x).sum / points.length

  c = params['coordinates'].first.first
  points.push GeoRuby::SimpleFeatures::Point.from_x_y(c.first, c.last)
  self.geom = GeoRuby::SimpleFeatures::Polygon::from_points([points])
end

To convert twitters json representation of a place into a WKRT geometry object in my database. This is cool, because then I can use some coffeescript like this to draw the polygon onto my maps.

if $place.geom.rings
  points = []
  bounds = new google.maps.LatLngBounds

  for point in $place.geom.rings[0].points
    coord = new google.maps.LatLng(point.y, point.x)
    bounds.extend coord
    points.push coord

  if bounds.toSpan().lat() == 0.0 or bounds.toSpan().lng() == 0.0
    marker = new google.maps.Marker {
      map : map
      position : bounds.getCenter()
    }
  else
    poly = new google.maps.Polygon {
      paths : [points]
      map : map
      strokeColor : '#ff0000'
      strokeOpacity: 0.5
      fillOpacity : 0.1
      fillColor : '#ff0000'
    }

That code also checks whether the polygon is degenerate (of zero size), and then plots a marker instead. Twitter always sends polygons, even if the polygon has zero area. Anyway. So this was all nice and took 15 minutes to code up. But it also let me do something super awesome:

def find_child_pois
  Place.find(:all, :conditions => ['kind = ? AND st_contains(?, geom)', 'poi', geom])
end

I now have a method to find all places contained inside a place. So for example, South of Mission in San Francisco, has Adobe Systems inside of it. You can also find out which places contain this one - for example:

>> Place.find_by_path('sfo/soma').find_parents.collect(&:name)
=> ["San Francisco", "SoMa", "California"]

I was pretty excited when all this came together so well. When I’ve got more time, I’d like to fix up the crawler that feeds twitterplaces, since at the moment it just dumps the database and recreates it every hour from a sampling of the twitter firehose.

Ah - for more good hours in the day.

Agile board in html5

2010-08-30T00:00:00-07:00

I put this together as a quick prototype of what an agile board in HTML5 could look like. I’m not a huge agile fan myself, but since I keep having to use pivotal tracker in projects - I’m always tempted to do a little bit more work on it and get it to a usable state.

Batched save ruby module

2010-08-29T00:00:00-07:00

In my work on twitterplaces - I’ve been commonly importing several thousand tweets at once from the streaming API. As I re-write my import process, I have to re-run this import job, so I’ve been trying to get it as fast as possible. Currently what I do is:

Use memcache
Use batched_save

Batched save is my module I created to let you include BatchedSave in a model, and then when you save a new record - it’ll batch up 100 records and use COPY to insert them. It is for postgresql only, and obviously only works in situations like mine where the data isn’t self-referential, but it’s mega fast.

My 10k submission was accepted

2010-08-27T00:00:00-07:00

Woohoo! Please go vote now! It’s a bit lame they’ve got the unsexiest screenshot first, but… yay!

My entry in the official 10k gallery

There is a community prize that you win by getting the most votes - so if you like my submission, please spread the word.

All done - in under 10240 bytes

2010-08-26T00:00:00-07:00

Well, it’s done. I just submitted my entry to the 10k apart competition. It’s been a pretty intense 24 hours, but I’m glad I gave myself some extra time this morning to polish up my entry.

Planning a trip to Paris, a screenshot from an earlier build

You can try it out here - all 9766 bytes of it.

Toolchain

I built the app using:

I have some extra code that is commented out, that does an autocomplete from Wikipedia when you add a new destination to your trip. So for example, you could type ‘Montmar’ and get an autocomplete for Montmartre. It’s nice because [a] I’m not very good at french spelling and [b] you automatically get some description text of the place from Wikipedia.

I also removed a column on the left that let you organise trips to different cities. I probably would’ve been pushing the 10k limit had I kept adding features, but mainly I didn’t have time to build and test the features properly. I enjoyed having time today to add some extra destinations to the trip, change the dates a little, and test the app thoroughly on each of the browsers.

Compression

I’d like to thank Nihilogic for their awesome canvas compression technique. Because I could compress the javascript and css so effectively, I didn’t have to muck around pruning bytes from my css and javascript, which meant I could spend more time polishing the app.

For reference - here’s the breakdown of the sizes of my CSS at different stages of my development pipeline. I used a simple bash script to compile, compress and package everything.

|              | Source | Comp  | Opt  | PNG  |
| Less CSS     | 7143   | 7589  | 5961 | 1898 |
| Coffeescript | 11388  | 13742 | 9445 | 3879 |

Internet Explorer 9

It’s been an interesting experience using IE 9. It looks like it’s going to be a good browser. I will still use Safari as my main browser and development tool (even if they did release IE9 for mac), but it looks like I’ll be supporting Safari 5, Chrome, Firefox 3.5 and IE 9 in all my future web apps.

Thanks

I’d like to thank @sminnee, @ristari, @nzkoz, @clarketus, @nikz, @agrath, @terrcin, @lancehodges, @youdoham, @nazdrug and @campbell for their support, feedback and bug checking while I put this thing together. I couldn’t have got it done without you all, thanks.

Why not consult?

2010-08-25T00:00:00-07:00

In the preparation for my leap back into the world of start-ups, I’ve been wondering why I don’t prefer to consult full-time. Consulting has a bunch of pros:

Excellent pay
Challenging projects
Turn your brain off at 5pm

It struck me today - that the main reason I don’t enjoy doing javascript / rails consulting is because I can’t do it all day long. I can do 4-5 hours of it and stay in flow, but after that it’s a real struggle. Sure I can churn out more code and fix bugs, but I’m not being creative, I’m overexercising the part of my brain that writes code.

And in a small business, that’s the perfect time of day to put away your text editor, open up skype, imovie and mail.app, and start down some of that marketing work.

I think this all comes back to Heinlein.

Specialization is for Insects

I want to be able to play the cello, climb a multipitch, fix a car, nurse a child, write a book, rally a crowd, run a marathon, pontificate on a beer, console a friend or any of a million more things.

Small business takes away your safety net, but it lets you try out as every act in the circus.

My 10k apart submission

2010-08-24T00:00:00-07:00

I haven’t had much time to look at the Facebook Places API, but I have been working on my 10k apart submission. This competition challenges you to build a functioning web app in only 10 kilobytes of javascript, html and css.

I’ve decided to take an idea I’ve had kicking around (a trip planner for weheartplaces) and see if it can be polished up to fit into the 10k limit.

Planning a trip to Paris, a screenshot from earlier this morning

Using the command ls -l *.css *.js *.html | awk '{sum+=$5} END {print sum}' to sum up the bytes, I’m already up to 12k of content for the unoptimized code.

I’m aiming to keep the entire project under 20k in unoptimized form, which will hopefully make the 10k limit achievable. Coffeescript and less output is quite verbose - and will benefit from closure and the YUI compressor.

Wish me luck. The deadline is lunchtime tomorrow (NZDT).

A varied gym schedule

2010-08-23T00:00:00-07:00

Like some web developers, I’m a bit ADD, so I find going to the gym a bit of a chore. Don’t get me wrong. I love the exercise, the clear headedness afterwards, the more energy you have in every day, the rest for your mind - a blissful 45 minutes where your brain doesn’t have to analyse pixels on a screen, or type strange patterns into the keyboard.

But spending 45 minutes with no mental stimulation - how dull is that! Especially when you go to the gym at quiet times (I don’t like noisy gyms), there’s not even anyone to talk to.

So my workout which is nice and varied is like so:

3km rowing ~12 minutes
Situps, pushups and chins x3 sets ~15 minutes
Treadmill on hilly mode ~15 minutes

I find the treadmill the hardest, since it’s just so dull - but I take it as a nice time to be all Zen (after the half an hour of exercise I find my inner Zen-buddha is easier to access). I set the treadmill to ‘randomly changing gradient’, so that your heart rate is constantly moving up and down.

Anyway - that works for me, and I’ve found it an easy workout to do consistently 4 times per week

Analyzing a places page

2010-08-20T00:00:00-07:00

Since Facebook places isn’t available outside of the US yet - I’ve only had two options for exploring the places api.

Firstly - my good friend loliver moved to Palo Alto two months ago to work for Wildfire, so I’ve been following his checkins.

Wildfire HQ Places page

I love the design, it’s nice and simple, has a strong social component (with the facebook wall), had a piece of content in the middle of the page (the map), some stats on the left to get an idea of the popularity of this place - and then a list of people who have checked in there which gives some ‘explorability’ to the places pages.

They don’t have ‘nearby’ or ‘similair’ places, but that might just be because it’s new and not enough of my friends have checked into multiple places.

Proxies and geolocation hacks

The other way I’ve been trying out the facebook is by proxying my requests through my US servers (which doesn’t seem necessary now that they’ve deployed the places functionality worldwide). The only downside is that when your browser tries to geolocate you, Facebook works out you’re not in the US and gives up.

The next trick for Monday will be to make a greasemonkey script that emulates the W3C geolocation API and let’s you check friends into Kentucky strip clubs.

Facebook Places

2010-08-19T00:00:00-07:00

I’m really excited about the release of Facebook Places. It’s not widely available yet but it should be rolled out to American Facebook users over the next 24 hours.

Facebook places on the iPhone

The features:

Checking in to places, for where you are at the moment
Tagging friends as being with you at a place
Seeing where your friends are and have been
An extensive API for accessing a users places data

It’s number 4 that’s exciting to me. Facebook gives a bunch of examples of ways the API could be used - including:

A travel application that gives people the ability to see which of their friends have already been to the place they are visiting.

That’s kind of exactly what I’ve been building. Starting next week, I’ll be working on the re-release of weheartplaces, integrated with the Facebook Places api.

We Heart Nokia

2010-08-18T00:00:00-07:00

I’m excited to announce that weheartplaces will be available for Nokia Series 60 phones, as well as the iPhone and Android.

I’ll be using the Nokia Webruntime (WRT 1.0), a browser-based environment that effectively allows you to create web/javascript programs that masquerade as normal symbian applications. I’ve just been testing out the functionality of my Nokia 6220, which is a $150 phone with 3G, GPS, driving directions and a 5 megapixel camera.

I got the icon going! That's like 80% of the work right there.

I’m not sure of the release date yet - but when it’s ready it’ll be available at my publishers page on the Ovi store.

To Market!

2010-08-17T00:00:00-07:00

Are you in Wellington and are serious about starting a tech business? I got an email from the bright ideas challenge with information about the following presentations, from various technical and business development peoples. I’ll be attending these three…

Strategy

Value proposition, business model, pricing model, finances and sustainability - Monday, 30 August 2010.

Technical

Intellectual property strategy, legal and governance, prototyping and funding options - Monday, 6 September 2010.

Go-to-Market

Go-to-market strategy, branding, exporting and distribution - Wednesday, 8 September 2010.

All events are held 4.30-6.30pm in the Memorial Theatre, Victoria University of Wellington. That marketing one sounds especially interesting.

Weird bug of the day

2010-08-16T00:00:00-07:00

On Safari, it’s nice to use ellipsis to limit the length of text. However, a warning to young players - text-overflow: ellipsis doesn’t work with webfonts.

Correctly functioning ellipsis stylingWhen using a webfont, Safari gives up, it's all too hard

Hey what does this button do?

2010-08-15T00:00:00-07:00

I’ve used photoshop for years and consider myself an intermediate user. You end up using photoshop a lot in webdesign to slice stuff up, or create decent logos - and prior to CSS3 it was my number one gradients tool. However - there’s a bunch of stuff photoshop doesn’t do.

With the advent of resolution-independent interfaces (retina display, next version of os x, landscape and portrait views of a site) and the need to create business cards, flyers, stickers and pixel art for websites - I’ve been learning Adobe Illustrator.

While playing with the weheartplaces logo - I discovered this menu item:

Hey - what does this effect do?Houston. We have a logo.

Logo Design

2010-08-14T00:00:00-07:00

I’ve been working on logos and design rules for Weheartplaces, creating dozens of Artboards in illustrator. I’m not a designer by trade, so some of my design is a bit stilted, but bear with me and eventually there might be a nice logo and style for the site.

Dax, probably too strong a shadow.Eurostile, going cold on this one.Oh, hai Steve.The Scott Pilgrim special, plus Illustrator gradients.For legal correspondence only.

Push state and Node.js

2010-08-13T00:00:00-07:00

I’ve talked with Sam Minnee (CTO at Silverstripe) a bunch of times about running jquery on the server side. It was only a hunch, I never knew exactly what the benefits would be - but with the availability of pushState on Firefox, Safari and Chrome - it suddenly makes a lot of sense.

What pushState and jquery on the server lets you do - is to have a pure javascript application that doesnt require page reloads to go to different urls. For example:

http://yourapp.com/
http://yourapp.com/search
http://yourapp.com/search/myterms
http://yourapp.com/tweet/12345

Can all be generated client side by your jquery and coffeescript code. This is an awesome way to write an app, you get a good responsive app and it’s easy to persist state between pages (eg your google map won’t spring back to the default location everytime you click ‘back’).

The number one downfall of javascript apps, is that there are now pages in your app that can’t be linked to - nor can they be crawled by google, or scraped by any other site that uses the http protocol. Building exciting new apps that ‘break the internet’ isn’t cool.

But if you can run your entire app on the server side - then if someone goes to http://yourapp.com/search/myterms - node.js can load your page on the server, recognize the url, run the correct javascript code to update the state as the browser would - then send it down the wire. So that non-javascript clients can access the same content that a desktop browser can.

It’s a whole new way to do accessibility.

Facebook places

2010-08-12T00:00:00-07:00

This analysis by Gigaom is interesting to me. I’m really keen to see what Facebook does with places. What they’ve done so far has been underwhelming technically, but has seen impressive takeup.

For example - try searching for Hashigo Zake (a Wellington beer bar popular with early adopters / geeks) on various services:

Facebook page - 413 people, 100s of posts
Foursquare - 65 people, 3 tips
Gowalla - 10 people, 4 comments
Google places - 5 maps - 2 reviews
Twitter - 4 tweets in the past week

Facebook has such massive uptake, that whatever it does has an impact. The understanding, technology and devices behind ‘location based’ services are so widespread that it’s really a case of Facebook just providing the tools and they’ll probably dominate existing location based services.

Also - note that all of the sites have reasonably terse (or even descriptive) URLs. All except Google. They really don’t get clean URLs.

Make LESS less painful

2010-08-11T00:00:00-07:00

I’ve been using LESS to make writing css for a jqtouch app less painful. However - the less compiler is a bit awkward to use by default - if you run:

lessc -g -w *.less

It recompiles your scripts everytime you make a change, which is perfect, and it notifies you by growl if you have a syntax error - wonderful.

However - every time you get a syntax error, it also blocks the terminal that you launched less from, and you have to go to that terminal and ‘Press [return] to continue…’. It’s pretty annoying. The solution is to load up /Library/Ruby/Gems/1.8/gems/less-1.2.21/ in your text editor, search for command.rb, and change the run! method, to disable the $stdin.gets and replace it as so:

# File has changed
if (not File.exists?( @destination )) or File.stat( @source ).mtime > File.stat( @destination ).mtime
  print Time.now.strftime("%H:%M:%S -- ") if @options[:timestamps]
  print "Change detected... "

  if parse
    # ...
  else 
    `touch #{@destination}`
  end
end

I also set the growl.priority to 1 in err(...) to make my notifications show up in red.

growl.priority = 1

You then get growl notifications on error, but simply correct the error and hit save again to recompile. No more hunting to find the right terminal.

CoffeeScript 0.9

2010-08-09T00:00:00-07:00

CoffeeScript 0.9 was released over the weekend. CoffeeScript is a compiler that turns .coffee files into .js files. CoffeeScript is a significant whitespace, rubyish language that makes large (and small) javascript projects much more manageable and fun.

I’ve been using CoffeeScript since 0.5, first with a multiplayer javascript game experiment, then more recently as part of my consulting with Lonely Planet Labs. It’s been interesting working with multiple developers on CoffeeScript - and seeing how it works in a production environment. I can heartily recommend it to teams that are looking to ease their javascript development.

CS 0.9 is a big change from 0.7.2, read this ticket for some of the background on the change. My own reaction to the change is interesting - I was against the change at first, and only changed my mind after 0.9 was released. I was impressed by how Jeremy got opininions and analysis from all the existing users, then quickly made such a fundamental change to the language, let people try it out in a branch - and finally decided to merge it back into stable.

All this - despite the fact that upgrading most coffeescript apps to 0.9 is non-trivial. Changing the symbol used for assignment will probably break most existing code. The assignment change was required to bring in yamlish object assignment which is awesome. The new release also has a host of other, non contentious improvements, around superclasses.

I think it’s a good reminder, that no matter how substantial a change - if you manage is right, and the benefits outweigh the costs, your community will adapt and embrace the new version.

Safari Extension

2010-08-08T00:00:00-07:00

I’ve been working through the extension authoring documentation for each of the browsers. I was excited when I got my little icon showing up in the Safari toolbar for the first time:

Weheartplaces toolbar button

Design features

2010-08-05T00:00:00-07:00

While redesigning this blog (I love being able to edit css and template as easily as blog posts, go github pages!), I found myself turning this:

Ordered chronologically

Into this:

Ordered by aesthetic

I find the wave a lot more pleasing to the eye, and I still kept the primary order mostly intact. This is one of my design tenets - lay out the page by weight and balance, then work out what text you can fit into each of the boxes.

It’s almost the opposite of how the signals do design, but I love it.

Brewdog Punk IPA

2010-08-04T00:00:00-07:00

I just discovered the awe inspiring flagstaff cellars, which is a tucked away little beershop in an unassuming corner of Melbourne.

The thing about this place though, is that is a stocker of Brewdog, Rogue, Epic and a number of good Australian beers - such as Matilda Alpha, Little Creatures and Moo Brew.

Anyway - they had this beer on special - Punk IPA by Brewdog, so I bought two and came home and tried them.

A very good beer.

The internet website rate beer said good things about the Punk IPA, although the haters at beer advocate weren’t so enamoured. The guy who drunk a Punk IPA that was 12 months past it’s best date - in particular - has a bad time with the beer.

My review of Punk IPA

This beer is awesome. It cost $5. I will drink it again. Probably in a few minutes when I get the other one out of the fridge. 5 stars.

Cities in the world..

2010-07-29T00:00:00-07:00

When you’re building an application for travellers, you come up against the problem of how do you get enough data in the system so that it’s useful for 455 cities with a population of more than one million people, 1054 with more than 500,000 and 2851 with more than 150,000 people.

Or to look at the problem a different way - the most visited cities in the world…

Most visited cities

This table at Wikipedia shows the top 10 most visited cities to be Paris, London, Bangkok, Singapore, Kuala Lumpur, New York City, Dubai, Istanbul, Hong Kong, Shanghai.

Crawling data for these cities

I’m not sure what the take away from this data is. Seeding 2851 cities with data seems very doable in a week of crawling, seeding the top 10 visited cities with deep data is also achievable - it’d be interesting to compare the list of top 2851 cities with the hometowns of the users of weheartplaces.

The real take away

Is that it’s better to encourage users to add content to ‘empty’ cities, than to expect to have every city to have a persuasive dataset when you launch your travel site.

Haversine in Coffeescript

2010-07-28T00:00:00-07:00

The Haversine formula is a equation that can be used to calculate the straight line distance between two coordinates on a sphere. It is commonly used by geohackers to work out how far in kilometres it is between two latitude/latitude coordinates.

Remember that lat/longs are actually spherical coordinates expressed in degrees.

This page at kapelica.hr contained a great implementation of the haversine formula in javascript, I tidied it up and ported it to coffeescript to measure distances to the nearest places in my iphone app.

Radians: (degrees) ->
  degrees /  57.2957795

Haversine: (lat1, lon1, lat2, lon2) ->
  R = 6371 # km
  dLat: Radians(lat2-lat1)
  dLon: Radians(lon2-lon1)
  a: Math.sin(dLat/2) * Math.sin(dLat/2) + Math.cos(Radians(lat1)) * Math.cos(Radians(lat2)) * Math.sin(dLon/2) * Math.sin(dLon/2)
  c: 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a))
  R * c

Note that the degrees->radians conversion isn’t exact, but the results should be within the margin of error for most GPS positioning systems.

Displaying only the first paragraph

2010-07-28T00:00:00-07:00

In redesigning my blog and publishing it on github pages, I wanted a simple way to show only the first paragraph of each post on the front page. The way I did it isn’t something I’m proud of - but it does show the power of css.

The following rule hides everything apart from the first child of a div.

div > * + *{
  display: none;
}

It works on safari, firefox and will work on ie9.

Offline manifests

2010-07-23T00:00:00-07:00

I’ve been working on an html5 app for Lonely Planet, and part of the task is to make the app work offline on the iPhone. Getting it to work isn’t too hard - you need to create a cache.manifest file and make sure it is served with the correct mime type.

The problem is online

When you have the phone offline and load the app, it loads in 2 seconds. When you turn the phone online again, it takes about 30 seconds to load all the javascript and so on.

It’s the caching rules

We have been deploying to google app engine, and by default - app engine sets all files to expire after 10 minutes. So even though the iphone has a local copy of the files, every 10 minutes, it’ll expire the local cache and redownload everything that the manifest refers too.

A checklist for fast online iphone apps

Use a manifest
Ensure the manifest is served with the mimetype text/cache-manifest
Ensure the manifest has no 404 errors in it
Ensure the manifest and all the content in it has an expiry date in the distant future
White-list your network resources explicitly in the manifest
Use jqt.offline.js to help debug the browser behaviour
Tail your webserver logs and keep an eagle eye for unnecessary requests

All going well, your html5 app will load instantly, and still be able to access network resources.

Coffeescript

2010-07-05T00:00:00-07:00

I’ve been doing a lot of coffeescript lately and can recommend it wholeheartedly.

That is all.

Skating in Melbourne

2010-06-20T00:00:00-07:00

I just went on a big skate of melbourne CBD.

I started at home (nähe Southern Cross Station), then skated out to Prudence in West Melbourne. Prudence is a highly recommended bar. The beer is expensive, but the cocktails apparently are affordable. From Souther Cross to Prudence is mostly uphill, which meant that after a few beers, a few conversations and a good laugh - it was all downhill to Section 8.

Section 8 is a very cool bar. During the day. I went there in the evening and wasn’t so impressed. The skate was good though - it’s mostly downhill from Prudence along Victoria Street - then turn right, drop down into town (it’s a pretty steep skate through traffic down to little collins street) - and a hard left into Section 8.

S8 only had one beer I wanted - and it was in what australians call a “tallie” (800ml beer). I took a few drinks - decided S8 wasn’t for me, then stashed the beer in my back pocket and headed through town. There were people everywhere (and you can’t drink in public anywhere in Australia) so it took 5/10 minutes before I got out of the city (skating past the Melbourne Cricket Ground by now) and could drink the rest of my beer. It was a lovely long skate past the MCG - down a shallow smooth slope on a path surrounded in trees and grass. Had some good tunes on - nice times.

Then I bombed from the MCG down to Richmond - eyeballed out the gig I was going to go see at the corner bar, decided it wasn’t worth $50 to go see - then caught a tram back to Southern Cross and mellowed along Spencer street to home.

Melbourne. Good for skating.

Merging identities

2010-05-24T00:00:00-07:00

I’ve set this blog up on Github pages using Jekyll. My old posts may find their way here….