Today I’ve been fighting a 10+ gigabyte mysql import. I’ve come across and solved a few problems that might be handy to someone in the future.

max_allowed_packet

In your /etc/my.cnf, it specifies the maximum packet size. I had the problem where some of the insert statements exceeded 1 megabyte in size, so increasing this limit to 64M solved my first problem.

Resuming from mid-way through an import

Next up, I wanted to cat the sql import, but skip all the tables up to table xyz. This I accomplished with a little bit of sed magic:

cat dump.sql | sed -n -e '/`some_table_name`/,$p' | mysql --init-command="SET foreign_key_checks = 0;" database_name

Note the set foreign_key_checks parameter to mysql, this is normally set at the top of your dump file, because we’re truncating the front off the dump file, you need to disable foreign key checks explicitly.

Using pipe viewer

PV is a tool to graph your progress. Instead of running cat and hoping for the best, you get graphical feedback on how long your database restore process will take.

I recently needed to be able to parse CSV export from excel, in Javascript. The only way I could find to parse it was a stream-oriented parser, so I came up with this that seems to cope with the majority of strangeness, including multiline cells, quotation marks, and escape and non-escaped cells.

It’s implemented in coffeescript:

class CSV
  constructor: (data) ->
    @raw = data

    @_parse(@raw)

  _parse: ->
    @rows = []
    row = []
    cell = ""

    offset = 0

    mode = CSV.RAW

    while offset < @raw.length
      ch = @raw.charAt(offset)
      adjacent = @raw.charAt(offset+1)

      if mode == CSV.RAW
        if ch == ","
          row.push cell
          cell = ""
        else if ch == "\r"
          @rows.push row
          row = []
          cell = ""
        else if ch == "\""
          mode = CSV.ESC
        else
          cell += ch

      else if mode == CSV.ESC
        if ch == "\""
          if adjacent == "\""
            cell += ch
            offset += 1
          else
            mode = CSV.RAW
        else
          cell += ch
      
      else
        throw "Invalid csv parser mode"
    
      offset += 1

    @rows
  
CSV.RAW = 1
CSV.ESC = 2

@CSV = CSV

This is a good example of code which will block the browser in a big-time way. For example, if you tried to parse a one megabyte CSV file, you’ll probably crash webkit mobile devices, and get script timeout errors in slower javascript implementations. Whoever, because all the looping logic is in a while, it’d be straightforward to make the parser run inside a setInterval loop, and let the browser remain responsive.

I’ve extracted the non-blocking .map and .select implementation that I use in my mobile apps to keep them responsive while processing large arrays. I called it overscore.js and it’s available on github.

As another part of my series on Javascript mobile apps, I’m going to discuss tooling.

I’ve listed my toolkit before, but here it is again:

• jQuery
• CoffeeScript
• Backbone.js
• Capt

jQuery

I use jQuery for the same reason everyone else does. It’s well implemented, fast and fun to use. The well implemented part becomes important when you’re dealing with fiddly implementation details of $.ajax calls. Being able to set headers, read headers and hook the progress events of the ajax object is important to a good mobile app. I use etags when checking for updates in my apps, specifying headers in a convenient hash shorthand is the kind of small detail that jQuery covers so well.

$.ajax {
  url : "http://example.com/endpoint/"
  dataType : 'json'
  timeout : 25000
  headers : { 'If-None-Match' : localStorage.getItem('etag') }
}

CoffeeScript

There are pros and cons of Javascript. The biggest downside I see is that if you’re working on projects that aren’t 100% coffeescript, you’re going to end up with ugliness. You either want to be able to convert all your code over to coffeescript (with the exception of 3rd party libraries), or you want to leave it all as javascript. Doing a combination is going to only cause heartbreak.

If you are sold on CoffeeScript though, it’s a fantastic tool. Using the fat arrow => to ensure class methods are called with the right this scope is an awesome timesaver. The biggest problem with CoffeeScript is probably debugging. This was easier in older versions of the coffee compiler, which had a 1:1 line to line conversion ratio, so model.js:23 was the same as model.coffee:23. Now, unhappily, your line numbers won’t match up, but as long as you can view the javascript output and find the error, you can easily work back and find out which line of coffeescript blew up on you.

Backbone.js

If you want a reason why to use Backbone, just view the apps using Backbone list on the backbone site. It’s a fantastic small piece of kit. It adds events, collections and models in a sensible way that will be familiar to any rails developer, but it does it without trying to pervert Javascript.

Basically, if you’re recieving json data from an endpoint and displaying it using Javascript, you should be using backbone or something similar.

I personally, used to make the mistake of trying to be ultra-minimal in my Javascript, avoiding doing heavy computation or having a lot of code over in the client. I’ve changed my way over the past year or two. If you’re struggling to do something without the correct tools, you’re wasting time, and you’ll probably end up with a half-assed solution. Refactor, break your code out into more classes, and increase your lines of code in Javascript.

Capt

This is my own tool, that I can’t really recommend to anyone else at the moment, since it’s a little bit half-assed and half-finished. But the important thing to do is to have a build tool that you can rely on. You want to be able to use .eco, .less, .coffee and .jst files easily, without having to compile them by hand, or having some makefile you run over and over. Find a build tool that watches your files, recompiles on the fly and includes each file seperately, so that when you get error messages, it’s not just error in bundled.js:31423423, which is going to be pleasantly impossible to debug.

I’d be keen to hear what other developers use for their build tools.

I’ve worked on three javascript mobile apps now, one used internally by Lonely Planet, one on the appstore, and another an unreleased site I built myself. I’ve been wanting to write up my notes on development for a while, so since it’s a sunny day and the house is tidy, I’ll put up some notes. First up…

Reasoning

First up, why even use Javascript + XHTML to build your mobile apps? The obvious answer is to get cross-platform support out the box, a well written app will deploy on phonegap to android and iOS with no extra work. Another answer, which is easily dismissed, is the benefits of having a single tool that you can use across more than just your mobile app.

Any business logic you encode in Javascript you can re-use for your desktop client and can be tested on the command line using node.js and jsdom. Where possible, one language is better than two in my book.

Finally, by using javascript, you are free from the tyranny of app-stores. I’ve had apps initially rejected by the appstore, and although they were easily resubmitted and are now selling well, if you’re doing anything “fringe”, that may run afoul of the appstore, having the option to deploy your app over the web as an html5 app is a nice fallback.

So, Google just released a static street view api, which is awesome, since showing a streetview of a place will usually jog someones memory much quicker than a top down view.

The only problem with the API (and it may be fixed in a future revision), is that you need to specify the heading that you want the returned image to be facing. This is a problem, since usually you only have the lat long of the place that you are visualizing, and not the bearing from the nearest road that a google streetview car drove along.

However - in my case, I have a complete set of OpenStreetMap data in my places database, so I could:

• Look for the nearest street
• Calculate the nearest point on the street
• Get the azimuth angle between the nearest point and the destination place
• Convert radians to degrees

And get the correct street view images. See this search for curry on Geonear to see what I mean.

Here’s the code I used (rails code using postgis):

<%= image_tag ".../streetview
  ?location=#{place.latitude},#{place.longitude}
  &size=280x200
  &pitch=15
  &fov=65
  &heading=#{place.street_view_heading}
  &sensor=false" %>

And in place.rb:

def street_view_heading
  street = nearest_street
  
  point = Place.find_by_sql(['select st_closestpoint(?, ?) as geometry', street.geometry, self.geometry])
    .first
    .geometry
  
  (Street.find_by_sql(['select st_azimuth(?, ?) as azimuth', point, self.geometry])
    .first
    .azimuth
    .to_f * 180 / 3.14159).floor
end

And we’re away. Very nice, thanks Google. :)

I needed some code for lerping google maps latlongs (for providing a custom streetview navigator). Here’s the function in coffeescript:

google.maps.LatLng.prototype.lerp = (b, i) ->
  lat = (b.lat() - @lat()) * i + @lat()
  lng = (b.lng() - @lng()) * i + @lng()

In the rankers app that I’ve recently finished for a client, I cached the users data offline, but needed a way to quickly and easy to see if the user had the freshed version of the data.

To speed up loading time, and enable offline access, I download the list of places (about 350kb) by an ajax call, and then cache the json and the etag of the latest revision in localStorage. Then when the app launches next time, I send a request for the newest version of the places list, and attach the etag to the request, so that rails can send a not-modified response, and save the users some data.

Here’s the rails pseudocode code I used:

@places = Proc.new do 
  Place.find(:all)
end

response.etag = @etag = [
  Place.find(:first, :order => 'updated_at desc').updated_at.to_i,
  Place.count
]

if request.fresh?(response)
   head :not_modified
else
  render :action => 'index', :mime_type => 'application/json'
end

And in the view:

<% cache(@etag) do %><%= @places.call.to_json %><% end %>

This cached the json (since json generation can be pretty slow in ruby), and sends the appropriate header. To handle this on the client side, I used something like this, to cache the places data, and the etag.

$.ajax {
  url : "http://example.com/some/endpoint"
  dataType : 'json'
  headers : { 'If-None-Match' : localStorage.getItem('etag') }
  success : (data, textStatus, xhr) =>
    if textStatus == "success"
      localStorage.setItem('etag', xhr.getResponseHeader('Etag'))
      localStorage.setItem('placesData', JSON.stringify(data))
    else if textStatus == "notmodified"
      data = JSON.parse(localStorage.getItem('placesData'))
    else
      throw "your toys"

    Places.reset(data)
}

I was going to implement my own version of etags and if-none-match, since I thought I might get heisenbugs with different implementations of xmlhttprequest on different mobile webkits, but in the end, I decided not to reinvent the wheel and use what already exists. So far this has worked well.

With my capt apps, I do most of the development and testing using the desktop safari browser, and then move onto testing on actual devices later on, once I’ve got the app mostly working. I use this fragment of code to detect if I’m running inside phonegap, or if it’s a desktop browser, and then instantiate my app appropriately.

function onDeviceReady(){
  window.app = new Application;
  app.start();
}

if(window.PhoneGap){
    document.addEventListener("deviceready",onDeviceReady,false);
}else{
  $(document).ready(function(){
    onDeviceReady();
  });
}

The window.app style of encapsulating my apps has worked well for me in apps I’ve built using capt and backbone, I still haven’t got to the point of encapsulating everything (all classes are copied into the window scope for example).

I’ve written a particularly gnarly query here that I thought I would share. It’s for iterating over a collection of points that have names, and identifying points that likely candidates for merging.

select
  name, floor(length(ST_ExteriorRing(st_box2d(st_extent(geometry)))) * 1000000) as length, count(id)
from
  places
group by
  name
order by 
  count desc;

Basically, it selects places with similair names, and returns the length of the bounding box surrounding those places. You can use this as a basic test to see if those places should be merged. There a tonne of situations in which this won’t work, but to get you out the gate, it might be helpful.

It must drive the google search team crazy, when they look at the sort of html the rest of the google organisation spits out. Take for example - this google places page. There is no html, no metadata, no microformats, no schema.org support. It’s basically impossible to index. How can they run the worlds biggest web crawler, but generate such obtuse and impossible to index code themselves? It’s astounding I say!

I’ve got two apps on the go using Phonegap, and this is just a quick post to say that I can highly recommend the mobile web as a development platform. My list of tools / libraries that I use:

• Capt for building the project and compiling / watching intermediate languages
• LESS.js for css
• Coffeescript
• Backbone.js
• Underscore.js
• My responsive map/reduce underscore.js helpers (yet to be released)
• Leaflet for touch-based maps that are GPU accelerated
• JSON.js from douglas crockford
• jQuery, although I may move to using zepto later on
• Phonegap for deploying to Android and iPhone
• Makefile for packaging and bundling
• Jasmine for speccing
• Latlon.js for distance and bearing functions
• iScroll.js for fixed position headers and footers
• Cross-origin ajax requests to allow local development while accessing remote servers
• ECO templates by Sam Stephenson for in-app rendering
• JST templates from underscore.js for static html

It’s a pretty nice toolkit. I need to do some more work on capt and put up a sample app so that other people can use my toolkit, but yeah, coffeescript, backbone and less.js are a great way to build mobile apps.

When you’re using xero and minutedock, you get used to having nice quality tools. It’s like having chisels that were shaped by master craftsmen. And then there’s your payroll package…

Payroll?

Payroll is the software that takes money from my company account, pays the tax man, deducts my taxes as an employee, then pays me in my personal account.

iPayroll

I’ve always used iPayroll. Giles Crisp is a great contact there (tell him Ben sent you), and provides great service. With iPayroll, you authorize them to move the money around and they get access to your company bank accounts. You will be paid every Tuesday without having to lift a finger. Plus it does all sort of HR stuff as well, so if you have employees, you can know know how many days off they should have.

Anyway - Ipayroll is great, but it’s kinda funky looking, and trying to sign up just recently has been a real PITA. Bugs a plenty with the sign-up process.

So I went to the xero payroll providers page.

Smart Payroll

Looks awesome. Until you actually look at the screenshots of the real app, not the pretty marketing page. It looks like it was designed in 1901. It makes iPayroll look a perfectly executed piece of modern art. Pass.

Flexitime

Looks awesome. The app is very xero / minutedock-ish, so I rang them up on their 0800 number and asked:

“Can I pay you money to take money from my account and move the money around?”

“Err - no sorry”

The Flexitime guy told me what I’d have to do to use flexitime to satisfy my obligations as an employer. Basically, set up an automatic payment, modify the AP as necessary, update flexitime, and remember to log into IRD once a month and upload a report that I download from the flexitime site. That’s a lot of remembering.

If Flexitime had put a ‘we manage everything’ option on their website with a nice CSS3 button for me to press and a well form that let me set up my organisation, I probably would have chosen them as my payroll provider. Oh well, back to iPayroll.

Here’s the non-blocking map reduce class that I was talking about. It uses at most 75% of the CPU, so that the browser remains responsive while your job is processed. You can destroy the job at any stage.

class MapReduceJob
  constructor: (inputsOriginal, mapFunc, reduceFunc) ->
    utilisation = 75
    inputs = inputsOriginal.slice()

    # todo - replace with a better clone function
    inputs = for input in inputsOriginal
      input

    outputs = []

    @interval = setInterval( =>
      tzero = (new Date).getTime()
  
      # Process at least one
      if inputs.length > 0
        outputs.push mapFunc(inputs.pop())

      # Process more
      while ((new Date).getTime() - tzero < utilisation) and (inputs.length > 0)
        outputs.push mapFunc(inputs.pop())
  
      if inputs.length == 0
        clearInterval(@interval)
        reduceFunc(outputs)
    
    , 100)

  destroy: ->
    clearInterval(@interval)
    delete @interval

You use it like this:

new MapReduceJob(
  ['a', 'b', 'c', 'd'],
  function(input){
    // do something computationally expensive here, like uppercasing the input
    return input.toUpperCase();
  },
  function(outputs){
    console.log(outputs);
  }
);

This is excellent for things like clustering markers on a map, or doing a backbone filtering on large collections. You need to structure your code to be asyncronous.

I’ve got a few things I might do with this code:

1. Port underscore.js to this continuations style, so that you can use all the existing functions. I’ve already ported the collect and inject functions.
2. Add a progress() method to the class.
3. Add support for webworkers for desktop browsers.
4. Port backbone.js to use these continuations, so that for example, the sort function doesn’t block when sorting large arrays.
5. Assuming the mapFunction is constant time, measure the number of inputs to process per interval, then stop checking the date function.

It’s pretty interesting stuff, we’ll see how we go.

Doing mobile development on your Android but can’t get any debugging action going? Add this code (from the inimitable Tom Shelfer) to the rails controller that provides your json views, so that you can serve your app on a different port than you serve your API:

before_filter :cors_preflight_check
after_filter :cors_set_access_control_headers

protected

# For all responses in this controller, return the CORS access control headers.
def cors_set_access_control_headers
  headers['Access-Control-Allow-Origin'] = '*'
  headers['Access-Control-Allow-Methods'] = 'POST, GET, OPTIONS'
  headers['Access-Control-Max-Age'] = "1728000"
end

# If this is a preflight OPTIONS request, then short-circuit the
# request, return only the necessary headers and return an empty
# text/plain.
def cors_preflight_check
  if request.method == :options
    headers['Access-Control-Allow-Origin'] = '*'
    headers['Access-Control-Allow-Methods'] = 'POST, GET, OPTIONS'
    headers['Access-Control-Allow-Headers'] = 'X-Requested-With, X-Prototype-Version'
    headers['Access-Control-Max-Age'] = '1728000'
    render :text => '', :content_type => 'text/plain'
  end
end

Then install the MacOS version of weinre and edit Content/MacOS/Launcher like so:

java \
    -XstartOnFirstThread \
    -classpath $BASEDIR/weinre-ui.jar:$BASEDIR/weinre.jar:$BASEDIR/swt.jar \
    weinre.application.GUIMain --boundHost -all-

Then launch weinre (webkit inspector for mobile devices), and add weinre to your web app and reload the browser.

<script src="http://192.168.1.77:8080/target/target-script-min.js#anonymous"></script>

Sorry this post isn’t very cohesive - but that there is a fantastic stack for debugging standalone apps to access existing rails apps. I’m using it to debug my phonegap version of the rankers app.

I’m clustering 1200 points on the iphone in javascript in an app I’m writing at the moment. On my iphone 2g it takes 1500ms in a worst case. There are two fixes I’m going to do:

Quadtree

I’ll probably end up implementing a quadtree, or a grid-based acceleration structure, since currently the complexity is currently n squared in a worst case. I’ll let you know how I get on with this.

Break something expensive

This is a generic pattern you can use in javascript (here implemented in coffeescript) to maintain responsiveness in your jobs. You create an interval that runs every 100ms, and it will do at most 50ms of work in each loop, until your data has been processed, then terminates the job.

inputs = [.., .., .., ..]
outputs = []

interval = null

interval = setInterval( =>
  tzero = (new Date).getTime()

  while ((new Date).getTime() - tzero < 50) and (inputs.length > 0)
    input = inputs.pop()
    
    outputs.push(
      doSomethingExpensiveWith(input)
    )
    
  if inputs.length == 0
    clearInterval(interval)
    
    console.log(outputs)
, 100)

I might pull that code out into a generalised form (since I could use it throughout the app) and also add support for chaining inputs and outputs:

inputs.mapReduce(filterFunction).mapReduce(processFunction).mapReduce(outputFunction);

Something like that would be pretty cool, but the above code is the simplified version that works for me.

)

I’m working on an iphone app that needs 650kb of json data to get started, so I needed a bit of progress display while the data is being fetched. This works on mobile webkit:

interval = null

$.ajax {
  url : "endpoint.json"

  dataType : 'json'

  xhr : () =>
    xhr = jQuery.ajaxSettings.xhr()
  
    interval = setInterval( =>
      if xhr.readyState > 2
        total = parseInt(xhr.getResponseHeader('Content-length'))
        completed = parseInt(xhr.responseText.length)
        percentage = (100.0 / total * completed).toFixed(2)
      
        console.log "Completed #{percentage}%"
    , 50)
  
    xhr

  complete: ->
    clearInterval(interval)
  
  success : (data) =>
    alert(data)
}

It’s a bit painful because of the way you access the raw xhr object in jQuery 1.5, but works nicely. I then store the data in localStorage (as a bit json blob) and we’re away.

I’ll be presenting at Wellington.js on the 13th of July about Backbone.js and my experiences with Capt, my javascript build tool.

That is all.

I was quite taken with these isometric minecraft renders, so I decided to dig out some of my old game code and try and build an isometric renderer…

That is created using webkit transforms, the minecraft textures. The heightmap is some perlin noise that I same via getImageData. Now to add some collision detection so you can walk around the world.

Four weeks on from my original post, I wanted to do a follow up about the “Day of Disconnection”.

The “day of disconnect” is now one of my favourite things. I look forward to it all week. It’s the day you don’t get sucked into the computer, and you have no excuses to go outside and do all the wonderful things in the world. And when you don’t feel like going out, you are forced to go out and find something fun to do.

My first Sunday was a day of drinking, but the following Sundays have been a lot more productive. Most recently, my partner and I drove up to the east cape of New Zealand in a little convertible and walked up to the easternmost lighthouse in the world. Another Sunday was spent exercising and collecting driftwood for a big bonfire out the back of the house, and I spent another Sunday doing some vigorous gardening.

So - I recommend you try disconnected Sunday. It’s a truly wonderful thing, and makes your Monday coffee + internet fix even better.

At this rate I’m going to end up making a nicely designed sundaydisconnect.com site. Hah. :)

I use Jasmine to spec / test my Backbone apps. It can be a pain setting up a Javascript testing environment (unless you’re using capt in which case you get a testing environment for free - but I digress. I decided to pull out some code from an existing app of mine and document a bit of it.

This isn’t best practise stuff, I’m making this up as I go along, but it is the summary of several years of Javascript development on desktop and mobile.

Start test suite:

describe 'places controller', ->
  describe 'search view', ->

I like to nest my suites so that any errors point to the correct controller and view.

beforeEach ->
  window.app = new Application
  window.app.location = { latitude : -45, longitude : 170}

My current app has a bit of an awful hack in that it uses a global ‘app’ instance that wraps up all the application functionality. It is easy to work with, but I don’t like it for testing, since your tests inevitably have side effects. But by recreating a new ‘app’ instance for each test you help defeat some of those side effects.

it 'should handle the truth', ->
  expect(true).toBeTruthy()
it 'should exist', ->
  expect(SearchView).toBeTruthy()

Some sanity checks to make sure the view we’re testing exists, there are no syntax or dependency problems.

it 'should instantiate', ->
  x = new SearchView { collection : new PlaceCollection }
  expect(x instanceof SearchView).toBeTruthy()
  expect(x instanceof Backbone.View).toBeTruthy()

Now we start to do something useful, by instantiating the SearchView. Notice that the view can be instantiated outside of the app, and without a specified element. Backbone will create it’s own div element for the view.

it 'should have render method', ->
  x = new SearchView { collection : new PlaceCollection }
  x.render()
  expect(x.el).toBeTruthy()
  expect(x.el.find('input').is('input')).toBeTruthy()

Now we can start to do some ‘functional’ testing and make sure that the view renders up an input inside my view. This is where you can start to do some poke and prod testing, and make sure the result is as you expect it. Note that I’m running my test suite inside safari, but which something like jsdom, you could run this test suite entirely inside node.js.

it 'should render nearby', ->
  x = new SearchView { collection : new PlaceCollection }
  x.collection.refresh $fixtures.placesNearby
  x.collection.status = 200
  x.render()
  expect(x.el.html()).toMatch /harvard square/i

I use capt to populate my $fixtures object, so I can edit my fixture data in yaml or json, and then poke it into the view and check it renders correctly. This is the weakest selector I could use. Something like:

expect(x.$('a:first')).toMatch /harvard square/i

Might be better, or using the :content selector.

it 'should render searching', ->
  x = new SearchView { collection : new PlaceCollection }
  x.render()
  x.el.find('input').val("Kirkla").trigger('keyup')
  expect(x.query).toEqual("Kirkla")
  expect(x.el.html()).toMatch /loading.../i

You can also trigger keyUp / keyDown and other events using jQuery event object. Note that in my case, this will trigger an ajax call to the server, which must be mocked out on the client so that the ajax call returns immediately and fires the success callback. I use the built in mocking support (spyOn) that comes with Jasmine for mocking.

I’m writing this post from the above the wing of a 18 seater twinprop plane, with the sun going down to my left, setting over mount ruapehu, which looks to be getting a good coating of powder for this years ski season.

Last month I decided to put my German trip on indefinite hiatus, and I’m going to be honest, it was a hard decision. But now four weeks later and I’m starting to enjoy this life. I’m living in a lovely beach house with some good guys. Travelling to Wellington city regularly to get a fix of friends, tech and cafe culture. I’ve been working with a great group of hackers based in Soho, London - apart from the early it’s an excellent way to work.

The next few months aren’t planned out, but I know I’ll be in Wellington for some of July, house-sitting my sisters lovely new house in Newtown, and I expect there’ll be a mission down to the Southern Island - assuming it’s still there after this newest round of earthquakes :(. Finally, my accountant reviewed my situation and helped me set a goal for the next two years, so maybe I’ll be about to get a piece of dirt to call my own in a year or two.

Anyway - hope everyone is healthy and happy and not persuing their dreams at the cost of their friendships.

Nick at Rankers turned me onto a new movement. It’s disconnected Sundays. I’ve raised it with Sam Minnee from Silverstripe. The goal is to not do any work, or to use the web on Sundays.

Basically, avoid using your computer or iphone at all costs. It freaked me out the first Sunday - what am I going to do to entertain myself? Obviously you can just read books. Or if you feel the need to work on your project you can draw your UI plans with pen and paper - but whatever you do, as long as you’re away from the computer, it’s guaranteed to be a good day, and you’ll come back extra charged up for Monday.

Anyway - so here’s to being unavailable on Sundays and spending it with friends, or with your mountain bike, or working on the garden, maybe jamming in the back yard with some guitars and a drumkit.

A common mistake people make in Backbone apps, is writing a view that assumes that data exists in a collection, so that when the user reloads the page, the collection is empty and the view won’t render.

For example, you have a Posts collection and a controller:

class Posts extends Backbone.collection
  # ...
  
class PostsController extends Backbone.Controller
  routes :
    "" : "index"
    "topics/:id" : "show"

  index: ->
    Posts.refresh {
      success: ->
        new PostsView { el : $(body), collection : Posts }
    }

  show: (id) ->
    new PostsShowView { el : $(body), model : Posts.get(id) }

The problem with this, is that when you reload the page and you access the index view, it fetches the data with refresh(), then waits until the posts collection has been fetched and renders the view. (This is a bit of a contrived example, but it demonstrates the behaviour).

After the collection has been populated, you click on a post, and it redirects to #posts/12345, a post id that is in the collection. All good.

The problem is when you hit command-r and reload localhost:3000/#posts/12345. The show view will be called straight away, and the posts collection was never populated.

GetOrFetch

I like to fix this with a little helper called getOrFetch:

class Posts extends Backbone.collection
  url: "/posts"
    
  getOrFetch: (id) ->
    if @get(id)
      post = @get id
    else
      post = new Post { id : id }
      @add post
      post.fetch()

    post

Let’s be honest, this isn’t magic, but it means you can replace your show action with:

show: (id) ->
  new PostsShowView { el : $(body), model : Posts.getOrFetch(id) }

Which means that no matter how you get to the show action, your model will be loaded from the server, or just use the local version.

Status

I’ve taken to using http status codes to represent what the status of a model is. For example a model that has been initialized but not loaded is status 0, if it is being loaded it is status 100, if it is loaded it is 200, if there was an error or the id was not found, 500 or 404, etc. This means that your view can have some code like:

<% if model.status <= 200: %>
  Loading...
<% else: %>
  <%= model.escape 'name' %>
<% end %>

Doing this in the mobile app I’m working on at the moment gives a nice ios-ish interface. To make your model status update:

getOrFetch: (id) ->
  if @get(id)
    post = @get id
  else
    post = new Post { id : id }
    @add post
    post.fetch {
      success : ->
        post.status = 200
        post.trigger 'changed'
      error : (e) ->
        post.status = 500
        post.trigger 'changed'
    }

  post

I’ve moved out to the east coast of New Zealand to work onsite with a client. I’ve been gardening, programming, brainstorming, prototyping, surfing, swimming and mountain biking. It’s been a nice time.

I’ll probably be in Wellington for the month of July, but apart from that I’m free to move around. If you have any interesting Javascript work on later in the year, I’d be keen to do a 2-3 month stint onsite. Anywhere in the world is fine.

I’ve been doing some research and development around offline mapping for a client of mine. Here’s my notes on the current state of technology.

Tile based offline mapping

The easiest way to do offline mapping is to use something like openlayers and store tiles offline. This is kind of gross at first glance, but in practise it works well. It’s very simple, the technology is mature, and above a certain zoom level, it can actually be smaller to store the raster data than the underlying unprocessed vector data. However, it clearly doesn’t work for high zoom levels.

To map an area the size of New Zealand using offline tile caching to zoom level 15, a naive implementation would take about 25 gigabytes of storage. With a little development you could probably get it down to 1-2gb, and with some serious development (adaptive zoom levels based on feature detection in the underlying vector data) you could maybe get to a ~200mb download, but it’s clearly not a trivial download.

However - this isn’t such a big problem if you’re doing a small area like a city or a town, So - assuming you don’t need ultra-high zoom levels, and you can solve the issues of download size, what technology exists to do offline mapping?

Delivering offline maps

The team at mapbox have created a suite of tools for offline mapping. Maps on a stick is an interesting tool that uses openlayers, firefox and html5 to create an offline mapping tool. It is written in javascript, and requires that you download the map tiles and store them on your local filesystem. The downside to this is that copying over 800,000 5kB tiles takes approximately forever. To solve this issue on their ipad app, Mapbox created the mbtiles format, which is involves putting all the maptiles into a sqlite database that modern filesystems can move around much easier.

Sadly, maps on a stick doesn’t support mbtiles, so I looked into what would be involved in adding mbtiles support to maps on a stick. Basically, it boils down to creating a firefox xpcom component (in javascript) that uses the firefox sqlite interface to expose the tiles to openlayers. It’s doable, but fiddly.

On the plus side, you could then wrap up maps on a stick, your mbtiles components and xulrunner to create an installable offline mapping tool. From my experiments with xulrunner and reading through the xpcom documentation, this is non-trivial.

Chrome store?

You could just shortcut all the xulrunner / xpcom stuff and do your offline apps using the chrome app store, or safari html5 offline support. I prototyped this and made a phonegap-powered ipad app that downloaded tiles from cloudmade and stored them in the html5 web database store. The downside to this is downloading the tiles is sloooww, and although you can do multiple requests at once using multiple subdomains, you still get slower throughput than if all the tiles were globbed together into one sqlite database.

Vector data and C++

The best solution, if you were keen to do some serious development, would be to bust out your compiler and package up mapnik or another renderer to act as an xpcom component for firefox. You could still serve tiles, so you could use openlayers (or tile5 or leaflet), and you would have infinite zoom, nice small vector data (the OSM dataset for New Zealand is about 25mB) and you would win all the technical brownie points.

Down this path also lies writing your UI in QT (ala Google Earth) or WX instead of using Firefox, so you could create a completely custom mapping tool.

Analysis

I haven’t finalized my decision yet, but it seems to me that there are no natural inflection points on the cost/benefit graph for offline mapping. If anything, I might lean towards xulrunner + something like “maps on a stick”, but it’s neither here nor there.

Basically, there are no good offline mapping solutions for netbooks / laptops, so maybe there’s a market niche there.

I’ve opensourced the repository holding my tile-based game engine. I worked pretty solidly on it for about two months (and put a bunch of videos up at reddit.com/r/hutch/), but it’s become a back-burnered project, so I’ve donated it to the html5 gamedev community.

The game was written before i developed capt, so all the code is located in public/scripts/ and hosted by rails. It’s designed to work with the cutegod tiles, and it’s probably a real PITA to get the thing set-up and running. I’d like to extract the coffeescript source and convert it into a capt project, so that it’d be easier for people to get the source up and running (and also start adding specs for everything), but don’t hold your breath on that one.

Orthographic 3d

The projection is orthographic, with y and z sharing the same axis (depth is faked using the z-index). The engine is designed in 3d, so positions are expressed as 3-vectors. Collision detection is done in the player model - if I started work on this again, I’d probably go for an architecture where classes have components, instead of a o-o big hierachy tree.

Multiplayer

There is some websocket multiplayer support in the game. The ruby-based server was basically just to prototype and experiment with realtime multiplayer in the browser. In practise I’d probably use a node-based server so that you didn’t have to duplicate code from coffeescript to ruby (for example the vector class, player class, etc, etc).

Future

I’m still very interested in multiplayer games in the browser. I’d love to build something like minecraft, second life or terrarium, where it’s basically a sandbox for people to build worlds and gadgets for other people to explore - but it’s obviously a massive project, and if I did more work on html5-gaming, it’d probably be as part of a team.

I got halfway through a tutorial on capt, my build tool for single page javascript applications using coffeescript, backbone.js, eco and less. The tutorial is available here, but it kind of stops halfway down. Feel free to fork the tutorial on github and submit patches, otherwise I’ll finish it up when I have time.

I joined the Greens party in the afternoon, met the leadership in the evening and then was interviewed by the film crew at the backbencher - because I support the Wellywood sign. Video below. It was a fun night.

I give you a time of day, you give me a name for the appropriate meal. I was trying to work out how to do this with a case / switch statement - then came across this syntax which is kinda nice:

def time_of_day_to_meal_name(t = Time.now)
   {
     0..5 => 'Midnight snack',
     6..10 => 'Breakfast',
     11..14 => 'Lunch',
     15..16 => 'Afternoon tea',
     17..21 => 'Dinner',
     22..24 => 'Late night snack'
   }.find { |k,v| k.include? t.hour}.last
 end

What would you like for lunch?

I’ve recently had a project morph from an online html5 mapping site, into an application that needs to run offline on netbooks and macbooks. So I’m investigating taking an openlayers, backbone and coffeescript application and packaging it up with Firefox webrunner (or Prism) as an installable package for Windows and os x.

Doing installable software isn’t new territory to me. At Indigo we built installable packages for Windows, OS X and Linux - so I’m pretty happy to work my way through innosetup or nsis, and the os x packagemaker. For the map tiles, I might try and adapt the mbtiles specification from the guys at mapbox. This will be a problem, as the database will have to be jerry rigged into place where Firefox can see it. I’m avoiding doing any binary firefox extensions for this project, although if things go well, there is the posibility of storing vector data locally and creating a mapnik build that operates as a firefox extension.

Data will be stored locally using backbone and sqlite, with regular online syncing to the official point-of-interest repository. So far we only have the html5 protoype done, so there is a lot of ground to cover - but I’ll try and keep you all updated on how things go.

This post comes as a surprise to me. When I planned to visit Brisbane for 8 weeks to spend some time with my girlfriends parents, I didn’t expect the town would grow on me so much. My previous opinion of Brissie have been is as a cultural wasteland of chromed-up bars and old holdens.

How wrong I was. I’ve come to really like brisbane, and I think it may be one of the best towns to do a startup in Australia / New Zealand.

Great technology scene

There is a great network of javascript, rails and other technology groups. They meet regularly, have some exceptionally strong developers (tile5, sproutcore and several other good opensource projects are maintained from brisbane). If you’re boot strapping, there is good demand for web developers working on modern projects, so you can make a good packet consulting.

Support for startups

The complex of buildings along southbank are invaluable for people working from home or without a fixed office. The edge is a beautifully designed digital workshop where you can grab a couch and free wifi for a few hours. Next door is the national library which has hundreds of modern and beautifully designed workstations that are free. Both universities have modern study halls where you just walk in and grab a desk. Mobile broadband is much cheaper in Australia than in New Zealand, so you can work from anywhere. The weather is better than Wellington or Melbourne so you can easily work outdoors wherever you can find shade and somewhere comfy. You’re just over an hour from Sydney by plane, so you can head down and pitch the collection of investors there, without having to live in Sydney.

Low cost of living

This is a comparative thing, but compared to Sydney or Melbourne (the only other two cities that could pay comparably to Brisbane) the cost of living here is much lower. A room in a cool part of town (within easy biking distance of the city) will cost you about $150 / week, which is less than half of what you’d pay in Sydney. Food and transport is cheaper than New Zealand (better public transport), and houses are better designed for the weather (compared to Wellington / Dunedin) so your energy costs will be lower.

Great standard of living

This is the bit that really rubbed off on me. When you first get here it’s all chromed up bars, bouncers and awful franchised cafes. But Brisbane has some gems, and it has enough gems that once you link them all up you can have a really good quality life here. It’s not as epic a cafe scene as Wellington, and I miss some of the fast food in Melbourne (lord of the fries!), but Brisbane does really well.

The easy long term view

There are as many types of startups as there are types of people, but in the kind of “startup” I find myself working on (part time projects funded by occasional consulting gigs, where the project might turn into a business over time), Brisbane is a really good match. It’s basically a overgrown country town, with all the advantages of affordability and relaxedness, combined with excellent modern infrastructure and a burgeoning creative sector.

If you’re using Backbone and reusing the same div for all of your views (in a single-page iphone app for example), make sure you call .unbind() on the div when you call destroy the view, backbone doesn’t do it for you, and if you don’t unbind the div, you’ll end up with a whole bunch of events (which are using delegate so they hang around) being trigger at strange and inopportune times.

This is the kind of post which is going to ensure I never get a job with the Aspergers Crowd, but I’m going to put it out there - I sometimes program and design without thinking.

There’s two aspects to this. One is when you’re working on a hard project where the functionality is spread across several files and modules, and you need to do some rearchitecture. I’ll often sit down, work out how I want it to work, then zone out and just refactor without reading the code on the page. Old-school lispers are famous for this, they can work out when to refactor and where the code is unbalanced by looking at the shape of the code, due to all the parentheses. I do a similair thing, relying on syntax highlighting and compile-on-save to point out when I’ve broken something.

The second case is when you’re prototyping a project, and you have to force yourself to not overthink things, to prevent becoming an ‘architecture astronaut’ (abstracting everything away into a GenericFactoryFactoryGenereratorCreator), and instead just deciding ‘I want to to X, so I’ll copy this code here and just search / replace until everything works’. I’ve been doing a bunch of this lately with a backbone mobile project I’m working on. I’ve got about a dozen lines of code that I’m copying and pasting into every model I create. I should create a new base class, and inherit from that, instead of cargo-culting (which is error prone and tedious), but I’m not going to do that until copying/pasting becomes unbearably tedious. By refactoring prematurely, I’m bound to make annoying assumptions that I’ll end having to code around anyway. So don’t overthink it and just copy / paste away until the extraction to make is painfully clear.

Obviously both of these techniques only work in a well tested environment, and I can put aside some time the next day to go over what you did and identify the bits that were left out or done badly.

Oh - and there is obviously that third type of coding without thinking, which is when you need to get something done that you really don’t want to, so you grab a few beers and create variables named the_variable_which_is_a_number_and_gets_incremented.

Proof of work is the idea of getting a client to do some computational work that is expensive to perform, but cheap to verify.

For example - if you send a client a random number, and then ask the client to keep increasing that number until the md5 hash of the number starts with four zeroes, it will take on average 2^16 attempts to find the matching number. Then the client sends the number back to the server and the server verifies that the hash starts with 0000…

The idea is to make the computation expensive enough that it makes it less worthwhile for spammers to attack your system. It’s a nice alternative to a captcha because it doesn’t require any user intervention.

Example proof-of-work solver

var x = parseInt(Math.random() * 0xFFFFFFFF),
  doWork = function(){
    var count = 100, y, h;
    while(--count > 0){
      x += 1;
      y = x.toString();
      h = hex_md5(y);
      if(h.match(/^0000/)){
        console.log("Solved... " + x);
        return;
      }
    }
    setTimeout(doWork, 25)
  };

doWork();

This code generates a random number, then increases it until a suitable hash is found. The work unit is broken up into 100 tests at a time, with a timeout between subsequent work so that the browser doesnt warn about script timeouts. This runs fine on a modern browser, but I imagine it would nail ie6. For production code you’d probably want to profile the doWork function and adapt the work unit size to keep the browser responsive.

Native solvers

One of the obvious downsides is that any javascript implementation of a hashing function is going to be much slower than a native implementation, so if a spammer wrote a native solver for your proof-of-work, they would be able to trivially solve your work requests.

For example - firefox takes about 8 seconds to solve my proof of work prototype, firefox takes about 6 seconds, and ruby can solve it in 0.23 seconds.

Alternative work algorithms

Many hashes is a well known and provably computationally difficult proof-of-work technique, but given the problem that a browser will never be able to solve them as fast as a modern GPU, it would be worth investigating other work that a browser could solve that could not be solved faster by any other method, for example calculating the size of an HTML layout.

I’ve just pushed capt 0.5.3. Code generation works properly now generate controller and generate model, and the capt watch and capt server commands actually work (I was a bit premature announcing on Tuesday). I’m using this on two projects now and it has been a really nice toolchain.

Autogeneration of spec classes is good for maintainability too. I kinda wish I had been BDD-ing Buddycloud at the moment. That’s some complicated stuff.

I’ve worked on “local” for a really long time, it’s something that’s got under my skin. Whether it’s creating mapping engines, url hierarchy for places or analyzing aggregate user trends, I just really love it. I started a startup in 2006 doing local. We almost got hire-acquired up by Google, but didn’t, long story.

Are there any early stage local startups out there?

Like one or two technical people that are working on something cool and want another hacker on board? Something part time would be a great start for me, I’m happy to toodle along for a while on a cool idea while we work out to get traction.

Things I’m interested in:

Quora for local

People ask ‘where’s a good place to get a beer in wellington?’ and you reply “[Hashigo Zake] and [The Hop Garden]” and the UI pops up little boxes saying ‘Did you mean Hashigo Zake at 25 Taranaki Streeet’? And you click yes and it turns your square brackets into a link.

Reddit for local

People can click on places in a city (reverse geocode to buildings and natural features using a combination of google local search and openstreetmap gazetter) and each place has a discussion board. Going to a city level shows all the conversations going on in the city, sorted by recentness. Build a url scheme like /paris/la-cite/notre-dame. I’ve written a prototype of this.

Delicious for local

Write a bookmarklet that lets you bookmark a webpage, and it hunts out any geodata in the page and saves it to a map for you. You can use this to plan your trips by saving all the interesting looking webpages and then accessing the list of places offline on your iphone. I’ve written this bookmarklet already.

Greplin for local

The user signs in using oauth to foursquare, google my maps, facebook, gowalla and twitter. Index all the places the user has been and all the places their friends go. Use this to generate a ‘personal black book’ of places to go while travelling the world. So you go to berlin, fire up ‘GeoGreplin’ and it shows you all the places your friends want to visit / have checked in to / have saved to a map.

Twitter local browser

Suck down the twitter geo-firehose and plot all the tweets onto maps. Break the tweets down into cities and suburbs. Aggregate the tweets together (using some kind of 2d histogram) to show where the interesting stuff is in your city. People can discover people nearby and see what’s going on. I’ve built a prototype for this too.

Anyway - that’s the interesting ideas. Anyone working on stuff like this? Or any angel list types want to fund me to find a cofounder and build one of these?

There comes a time in every project - after days of prototyping, design, tweaking, rigorous speccing and bdd, after several deployments to staging and a/b testing with beta users, after polls and roundtables and whiteboarding - that you have to admit that the project isn’t any fun to use.

And that’s the opportunity to sit back and reimagine it, then make some small (usually not huge) tweaks until it starts to come together again.

It’s hard to discover new activity in Buddycloud, time to fix that.

Log into diaspora-x and there’s no indication of what activity is happening on your channels. Simon said to put the channel list on the left and bubble activity to the top. Let’s try.

span{
  display: inline-block;
  height: 20px;
  width: 20px;
  text-align: center;
  font-size: 0.8em;
  background: #999;
  color: white;
  border-radius: 10px;
}

Okay - add a litte badge showing the number of posts, and reorder the channel list by number of posts.

Nice.

I’ve been doing some work on buddycloud / diaspora-x user interface. Screenshots follow:

I have to admit that the buddycloud ui looks a lot nicer, with blues and colors, but I’ll be sticking with the grey and black ui that diaspora-x features.

I’m getting ready to deploy the latest diaspora-x.

I’ve resumed work on capt, since my diaspora-x codebase relies on capt, and although I expect brunch or cinco will take over as the build tool of choice for coffeescript / backbone apps - capt works for me now.

Capt is a tool for development and deploying javascript single page apps. It takes care of including all your javascripts in the right order, and compiling your static html pages. It has some support for jasmine, coffeescript and less. It’s very creaky, but my collaborators on the diaspora-x / Buddycloud web client needed to be able to use it, so I’ve packaged up capt as an npm.

So you can install capt with:

npm install capt

Once it’s installed, create a new app and serve it using the built in server.

capt new myproject
cd myproject
capt server
open http://localhost:4000/

And enjoy the capt goodness. I’ve put up a basic webpage explaining how to use capt. I imagine this release is pretty broken, it’s only really intended for diaspora-x development, but give it a go. I’ve been doing small commits on capt for a while now, so it’ll eventually turn into a decent tool I hope.

This is a bit of a contentious post (that commit is comedy gold) but I think people talking about rails using coffeescript as the new default client-side language are missing the point that coffeescript doesn’t complement rails, it supersedes it.

I come from a biased position, since I’m known for front end development, but I’ve been doing rails since version 0.6, and most of my contract gigs have a ruby on rails component. Rails is a great tool for building middleware, and when it was invented it was a perfect solution.

2006

Common technology was MySQL and PHP. Front end development was pretty low-impact, mostly you generated html on the server and sent it to the client, sometimes you sent html fragment back and forth for some ajax-interactivity.

Into this environment, rails was amazing. It was a great framework for building apps, gave you conventions on how to structure your app, forms and data models. And then rails came out with the ajax helpers and people went crazy for it. Over time rails added excellent json support and people started using it to create pure json api sites (like twitter does these days), but fastforward to 2011…

2011

We now have pure json databases (couchdb, mongodb), localStorage and thousands of public APIs. And any truly modern app will probably be written in pure javascript (and rendering their static html by a node.js instance for disabled browsers). Into this environment, Rails makes less sense. If you’re building a facebook add-in, you can probably do it entirely using FBJS. If you’re using the foursquare api you can do that in the client, same with twitter. If you need to do an app that stores tonnes of data - get a free couchdb instance from couchone. If you need “web scale” - you can use amazon simple db.

Rails doesn’t need to feature in any of this. And although rails will live on forever (like php does), and it’ll probably adapt to it’s new role as an authentication and permissions layer between the client and the database, I would be surprised if a better solution doesn’t come along. It could be a more mature CouchDB authentication / permissioning framework, or Postgres might release a json interface and people go back to writing stored procedures to authenticate / secure requests.

Middleware is becoming less important

I’m sure its just a cyclical thing, we go back and forth between thin client to thick client architectures, but it seems to me that rails might be on the downswing, to be replaced by some hot new tech that takes less work to create pure javascript apps.

Note that I’m not saying node.js is going to replace rails. I think there’s less need for middleware these days.

I’ve been designing today. The current Buddycloud site needs a refresh - so I busted out Jekyll (the tool I use to generate bennolan.com), some css3 and the blueprint css framework.

It came together quite nicely, and because we’re hosting the repository for the site on github - you too can go visit the site - even though it’s unfinished.

Which brings a dilemma. Do we make the repository private - remove the hosted instance and only require people working with Buddycloud to view the in-development site? There’s sure some interesting / awkward stuff in there - a lot of lorem ipsum text and some rough guesses of price points and a business model.

Well - to be honest, what kills most small businesses (which is what Buddycloud is at the moment) is obscurity or apathy, not giving interested parties too much insight into your business. So we’ll leave it up for now, and then everyone can see the site when it’s ready for ‘public’ deployment.

The Buddycloud web client is available on github. I started with the Diaspora-x codebase, and have been backporting in features from the codebase that was started by Stefan Maka (another Buddycloud contributor).

Using localStorage to accelerate load time

When you fire up diaspora-x, it takes about 15 seconds to finish loading the #home page. Obviously this isn’t going to play in a world where page loads should take less than 1000ms. Ideally I’d like the page to load in about 100ms. Assuming we can get everything cached locally, that gives a pretty generous time for the browser to cache and run the javascript payload.

The first part of the optimization was to prevent unnecessary requests to the XMPP server. Once you’ve requested all a users posts, there’s no need to re-request the same posts. We can use Result Set Management to only ask for posts newer than the last post we recieved, and then we can use backbone-localStorage to save the posts in localStorage in the current browser.

class ChannelCollection extends Backbone.Collection
  model: Channel

  initialize: ->
    @localStorage = new Store("ChannelCollection")

And then before we start the app…

Channels = new ChannelCollection
Channels.fetch()

Then remember to clear localStorage when the user logs out.

localStorage.clear()

Normally when a client rings up and wants you to build a replacement for Facebook, you smile politely and say that you’re full up for the next 18 months.

But, Simon from Buddycloud has asked me really nicely to build the front end for his distributed social network, so I said I’d give him 5 weeks and see what we can pull off.

Buddycloud

The buddycloud protocol is an extension for XMPP, the system that gtalk uses. Buddycloud takes the existing XMPP extensions (publish / subscribe, rosters, federation) and builds a social network around them. Buddycloud is an opensource protocol, Simon and others are trying to get the protocol in front of the xmpp standards committee.

There are three implementations of buddycloud-compliant servers at the moment, one for ejabberd, one for prosody, and a new one that uses node.js and plugs into any standards-compliant jabber server.

Web front end

Back in December, I created Diaspora-x, which was a first cut at building a social network on top of XMPP. It is a pure HTML5 app, just javascript, html and css - the backend service is provided by the XMPP servers (running ejabberd and prosody at beta.buddycloud.com.

This was an interesting experiment, and showed it is possible to build a responsive social network that gets all it’s data from an xmpp network.

Twitter, Facebook and Buddycloud

It’s interesting to note that new twitter uses the exact same engineering as Diaspora-x, in that when you visit the website, you’re actually loading a large javascript application. The tweets and profile data that you see is transferred afterwards by an XMLHttprequest.

The existing Diaspora-x client (which is the basis for the new Buddycloud web client) does the same thing, it’s a Javascript app, powered by Backbone, jQuery and Underscore.js - which instead of loading tweets from api.twitter.com, it subscribes to an XMPP channel over BOSH (XMPP over http protocol), and messages from your contacts are pushed down to the browser.

The current Facebook UI does a similair thing, it loads a small amount of static html when you load the page, then most of the rest of the page is constructed from Javascript running in the browser. (Facebook is a bit different because it uses their Primer tool that downloads html, instead of doing the templating in the browser - but similar concept).

Building the Buddycloud Web Client

The Buddycloud web client will be an Opensource project, so you’ll be able to watch my progress at github once the repository is up. I’ll try and blog regularly here, so watch for progress…

I put together this astar solver in coffeescript. It expects that nodes have a key() function that generates a unique id for the node, and an getAdjacentNodes() function that returns valid adjacent nodes.

It seems to work nicely.

Also - it’s my birthday today! Woo! :)

class AStar
  constructor: ->
    @openNodes = {} # List of openNodes nodes (nodes to be inspected)
    @closedNodes = {} # List of closedNodes nodes (nodes we've already inspected)

    # The maximum potential trip length we would consider
    @maxHeuristic = Point.origin().squareDistanceTo(new Point(5, 5))

  findPath: (start, destination) ->
    # g = 0 #Cost from start to current node
    # h = heuristic(start, destination) #Cost from current node to destination
    # var f = g+h #Cost from start to destination going through the current node

    start.f = @heuristic(start, destination)

    # Push the start node onto the list of openNodes nodes
    # openNodes.push(start) 
    @openNodes[start.key()] = start

    #Keep going while there's nodes in our openNodes list
    while @openNodes
      #Find the best openNodes node (lowest f value)

      #Alternately, you could simply keep the openNodes list sorted by f value lowest to highest,
      #in which case you always use the first node

      node = { f : Infinity }

      for key, n of @openNodes
        if n.f < node.f
          node = n
  
      # No nodes remain in openNodes
      if node.f == Infinity
        # No path could be found...
        console.log "No path could be found"
        return null
        # console.log @closedNodes
  
      # Check if we've reached our destination
      if node.equals(destination)
        path = [destination]
  
        while (node != start) # && (node.parentKey)
          node = @closedNodes[node.parentKey]
          path.push node

        path.reverse()
    
        return path
    
      # Remove the current node from our openNodes list
      delete @openNodes[node.key()]

      # Push it onto the closedNodes list
      @closedNodes[node.key()] = node

      # Expand our current node
      for n in node.getAdjacentNodes() when (!@closedNodes[n.key()]) && (!@openNodes[n.key()]) 
        # console.log(n.key())
        n.f = @heuristic(n, destination)
        n.parentKey = node.key()
    
        # Prevent really long paths
        if n.f < @maxHeuristic
          @openNodes[n.key()] = n
        # else 
        #   @closedNodes[n.key()] = n

  # An A* heurisitic must be admissible, meaning it must never overestimate the
  # distance to the goal. In other words, it must either underestimate or return 
  # exactly the distance to the goal.
  heuristic: (a, b) ->
    a.position.squareDistanceTo(b.position)

Here are the specs so you can see that it does actually work, but the require a bunch of other classes that I’ve been working on - and I don’t have time to tidy them up for release right now:

describe 'astar', ->

  describe 'square map', ->
    # Create a big square tile
    map = new Map
    bounds = new Bounds(Point.origin(), new Point(10,10))
    for point in bounds.getPoints()
      map.set(point, new Tile)
    origin = map.get(Point.origin())

    it "should work horizont", ->
      a = new AStar
      x2 = map.get(new Point(0,5))
      path = a.findPath(origin,x2)
      expect(path.length).toEqual 6

    it "should work vertical", ->
      a = new AStar
      x2 = map.get(new Point(5,0))
      path = a.findPath(origin,x2)
      expect(path.length).toEqual 6

    it "should work diagonal", ->
      a = new AStar
      x2 = map.get(new Point(5,5))
      path = a.findPath(origin,x2)
      expect(path.length).toEqual 11
  
      path = for tile in path
        tile.toString()
    
      expect(path.join(" -> ")).toEqual("0, 0 -> 1, 0 -> 1, 1 -> 2, 1 -> 2, 2 -> 3, 2 -> 3, 3 -> 4, 3 -> 4, 4 -> 5, 4 -> 5, 5")

    it "should several times", ->
      x2 = map.get(new Point(5,5))

      a = new AStar
      path = a.findPath(origin,x2)
      expect(path.length).toEqual 11

      a = new AStar
      path = a.findPath(x2,origin)
      expect(path.length).toEqual 11

      a = new AStar
      path = a.findPath(origin,x2)
      expect(path.length).toEqual 11

      a = new AStar
      path = a.findPath(x2,origin)
      expect(path.length).toEqual 11

      a = new AStar
      path = a.findPath(origin,x2)
      expect(path.length).toEqual 11

      a = new AStar
      path = a.findPath(x2,origin)
      expect(path.length).toEqual 11

    it "should work on a single step", ->
      a = new AStar
      x2 = map.get(new Point(1,0))
      path = a.findPath(origin,x2)
      expect(path.length).toEqual 2

    it "should work on origin->origin", ->
      a = new AStar
      path = a.findPath(origin,origin)
      expect(path.length).toEqual 1

    it "should return empty when impossible", ->
      map.set(new Point(100, 0), new Tile)

      a = new AStar
      x2 = map.get(new Point(100, 0))
      path = a.findPath(origin,x2)
      expect(path).toEqual null

For future reference, the against me nervous energies sessions are linked in this post.

My favourite band of all time. Make sure you also check out borne and most importantly, don’t lose touch.

Whenever I use OpenLayers or even Tile5, it annoys me that no matter how much you optimize your javascript code, your html5 map will never be as smooth and fluid as the native iphone maps app. I decided to try and fix that today.

This demo is only a proof of concept, but if you try it on an iphone (even my 1st generation iphone), you’ll see that it pans as quickly as the native client. It uses:

• A 60fps interval to update the map position
• Damped map velocity movement (flicking)
• Translate3d to use hardware acceleration
• Suppresses touch events and user-select events to prevent the user interaction slowdown
• Zepto.js for event management

I’m discussing with Damon from tile5 to get this code put into tile5 (since writing an entire javascript mapping library isn’t on the cards for me today), but I wanted to show the performance that javascript mapping libraries should be aiming for.

I was wondering what the smallest set of opcodes would be for a turing complete language.

Brainfuck seems to indicate that you need 8 opcodes:

• Move the data pointer left or right
• Increment / decrement the data pointer
• Write the data pointer to the output
• Read input to the data pointer
• Loop until the data pointer is zero

I’ve been thinking about AI a bit lately, and a question I’ve had, is what is the smallest non trivial programming loop you could make? What number of opcodes would it be? I guess it’d have to be a program that took evolving inputs and iterated to approach an optimal solution.

I’m thinking of a program that has changing inputs, so that there is no one optimal solution, the best the program can do is constantly seek to a solution. It’s a thought experiment, in that that you can calculate an upper bound for how quickly a genetic program could evolve by working out the simplest non trivial program, and calculating how fast it could be run on a modern cpu.