Mapping the product/market space – an hallucination

Imagine a vast multi-dimensional space.

Each point of it represents a specific need that a specific person has, an iota of utility.

The dimensions represent crazy things… Is the need in Africa or in Europe? Is the need on a LAN or on the web? Can travel to satisfaction of the need be done by public transport? How much does the need meet each of the basic needs? Is the need for shelter from the weather? Does satisfaction of the need require the skill of programming?

One night, someone persuades cartoonist Randall Monroe to take LSD so that he can draw a map – it’s a bit like this, only even for a rough map he has to paint it on the 26 dimensions of a bosonic string [1].

Bright spheres of product/markets swarm over the phase space.

Chain supermarkets hang in an oppressive morass, dominating one corner. The part of this representing best-selling novels scantily overlaps the impenetrable fortress to the north that is Internet book stores.

Extending from there to the south-by-south-west is where I’ve spent my working life hanging out. A bursting region, tearing the fabric of the space, loosely encompassing all those needs met by computers, by the Internet.

Little of it is very direct – it fires off in tendrils in the spin direction, which represents meta-needs, such as of a car factory needing a toolmaker needing a CAD software company to write some software to exactly position a metal drill [2].

Zooming in, zooming in, you find the dimension for needs met by the web, spanning the part for people who have credit cards (and are prepared to use them) and the part for people who want to share complex (yet free to reproduce) goods for free. It’s where ScraperWiki lurks. It looks a bit like this.

If you draw the boundary of a specific product, you’ll find it mainly covers a coherent lump of the space. It’ll have zaggy edges, wild spikes where someone uses it for something not quite intended – a tool intended to be used to track faults in software, instead used to track repairs needed to a house[3].

Consultants and salesmen work busily all over the map, stretching tendrils out from multiple products by hand, extending filaments from the bright areas into the dark.

By Jonathan Kos-ReadYou can see all three stages of innovation [4] bubble in the raw structure.

1) A bad idea for a product business pops into quantum reality and immediately gets sucked into the huge black hole of a nearby product that is very flexible. Elsewhere, someone spots a dimension nobody had considered clearly before, and finds a medium-sized dark spot nestling there. They have an idea that would fill it, but it’s too early, or they don’t have the time, passion or money in this life to light it.

2) Much later, a second person (maybe after talking in the pub to the sister of somebody who years previously read a tweet by the first person) has almost exactly but not quite the same idea. They make a prototype, and a brief glimmering star lights up in the darkness. For a few weeks it’s kindled, a concierge service lighting it by hand as a few of their friends try it out and find it useful. Or maybe they launch a product, but it doesn’t have the success they wanted. Too early, lack of time, of money, of passion, distraction by hedonism – whatever reason, not enough people buy it to make a viable enough business, and the fire is soon enough blown out.

3) Finally, a third person realises something important about the market of the new product idea and pushes it along the space to a fatter area. Anyway, time has passed and all the dimensions have reconfigured somewhat. Finally, a new glowing inferno burns in the gap.

A startup is by definition a hunt for one of those spaces, or a subspace within one, just the right size and place for the team, and the time.

[1] I can’t show it you – your browser doesn’t have the plugin.

[2] Randall briefly tries to sketch the supply chain lines on his map. The design of the original map was so tight, the reoriented mess now so incomprehensible, that he has to tear the whole thing up and start again.

[3] Someone else is keeping their DNA there.

[4] See the Myths of Innovation by Scott Berkun.

Heroku’s early history: 4 home pages that made $212 million

I decided to investigate Heroku’s early years.

You can learn a lot from even quite recent tech history (see my previous article on version control).

My tool? The Internet Archive. It’s an elephant that never forgets your pivots.

1. November 2007 – code in the cloud

Ruby on Rails is riding high. But impossibly hard to deploy.

Y Combinator startup Heroku’s first home page (see right – apologies for the lack of images, which has not recorded) screams onto the web.

It’s remarkable just how much it did from the get-go.

Not only did it solve the Rails deployment problem (“Instantly live!”), but you could also write the code in your browser for the first time (well, since Zope!) (“Create and Edit Online”), and share it with others (“Share and Collaborate”).

Really, hand on heart, until you read that last paragraph, did you remember that Heroku had an online code editor? And that was a major part of their initial value proposition? And what’s this about sharing?

Cleverly, they’d realised people will probably be worried about lock in (“Import & Export”). It’s not clear from later home pages that they really were.

No mention of scaling. No origami.

2. February 2008 – it’s a Rails platform

By February, Heroku had its debut on TechCrunch (including screenshot of their online IDE).

The home page (see later version of the same with images) talks more about deployment and scaling and git, with still a frisson of “coding in the browser”. Nothing about “sharing” any more.

It’s illuminating to read the comments on the TechCrunch post. As ever, full of nay sayers, from the ignorant …

Anyone who thinks Rails is difficult to configure doesn’t know what they are talking about. “Heroku = Fail”

(Rails was at the time very hard to configure on the server, I spent days struggling with hacks to cope with dieing and leaking Mongrel instances, at a time when mod_php just worked.)

… to the preternaturally informed, as you shall see:

Hosting isn’t trivial, but any solution that starts out “OK, give up your editor and terminal and version control and offline editing” seems pretty fishy for anything beyond the 15-minute blog demo.

3. February 2009 – fixed width becomes hip

A whole year passes until Heroku change their home page (see later version of the same with styling) again.

Several radical changes. For the first time ever (well, maybe since the early days of the web), actual code appears on the front page of a fashionable website. And not just Ruby or Python code, shell code.

My explanation for this suddenly being hip again, is the resurgence of the Mac around this time. OSX makes accessible to people who five years earlier would have used Windows and thought the command line was for dumb retrobates (sic).

What a home page! All shiny and black with a fixed width font, and $ signs. Wow.

Secondly the value proposition has simplified to three points. 1. Instant ruby deployment. 2. Git. 3. API. The latter two were merely mentioned in passing a year before, now they’re a key part of the value.

Github launched the previous April and at this point has 45,000 registered users and is on an exponential growth curve. Not too surprising that Heroku give git higher billing this year.

The most radical change is what is missing. Have you spotted it?

Tucked away at the bottom of the page is a tantalising link. “Heroku Garden Transition info“. WTF? A blog post “What’s up at Heroku” explains. They’ve completely rewritten it.

We also learned what our users want from a commercial version of the platform, and surprisingly to us, we discovered that there aren’t just a bunch of features we need to add, but some we need to remove as well (platform features often involve trade-offs).

Having made that discovery, we knew we needed to create a second version of the platform, to pursue exactly those requirements, designed from the ground up (with our hard earned knowledge) for commercial production use.

Features removed?

Indeed. No more online code editor. No more sharing. That’s all hived off to “Learn Rails, Work from Anywhere, Move to Heroku Later” is the value proposition of that new (erm, old) app nursery.

4. October 2009 – doubling down, adding on

In the autumn, it changes again (see later version, much the same with styling). Even more code, more clearly showing you that the commands are everything you need to get started. It’s the geekiest home page ever.

That said, there’s a much better split of the page; the value proposition for people who don’t speak /bin/sh is now laid out clearly on the right.

Again, the most important thing about this new home page is hidden away at the bottom (at least in the later 2010 version, it’s commented out in the HTML of the original 2009 version).

“Add-ons”. Despite throwing away features with their rewrite, an absolute corker of a big new feature appears. It’s how Heroku becomes effectively “an app store for systems administration”. But that’s another story.

This final home page holds them out well, lasting beyond the next December when Salesforce buys Heroku for $212 million. Cash.


If you love the origami, and are wondering when it appears, that is as late as June 2011.

And Heroku Garden?

In January 2010, just three months after its debut in October 2009, it’s axed. Poor thing.

I like to think that one of the founders of Heroku shed a tear. That they’d spent at least a year arguing until they were fit to explode just how important the online code editor was, how a “YouTube for web app sharing”, a “Google Docs for coding” was the next big thing.

And then watched the relentless figures, as everyone just used the deployment system, ignoring the editor, ignoring the sharing. Silly, it was the “app store for sysadmin” that won it.

For the record, I’ll wager that in 10 years time loads of people will be coding in the cloud. In their web browser, and using apps on their tablets. It seems crazy right now, but you’ll see. It seemed crazy 10 years ago that we’d all be using an online office suite.

Meanwhile, remember the lesson. Even the best don’t get everything right straight away.

Do several things well. Double down on the ones people like.


Products and markets are the same thing

At mySociety‘s annual retreat I gave a lightning talk about how I’ve come to realise that products and markets are the same thing. I’d originally intended it to be a blog post, so here you are.

It’s a story of geeks learning.

What is a product?

It really is magic.

Before the invention of products (whatever that means!) you had to make bespoke things. You want a tool, you have to bash the rocks together to make your own custom tool.

What’s magic about industrialisation, or copying computer software, is that you can design something once and lots of people benefit from it. The design, which is to say the finding of that commonality of use, becomes much more expensive.

Fred Brooks in the Mythical Man Month describes this for computer software.

A Program is the object an individual uses in estimating productivity, it is ready to run in the programmer’s environment.

The Product can be tested, repaired and extended by anybody, is usable in many environments for many sets of data. It is generalized and documented, and costs at least 3 times more than a Program for the same functionality.

Making good products

I wrote earlier in the year about how “geeks are now good at usability”. What I really mean is that there is now a reasonably strong culture of making quality products. Ones that are designed for purpose of the user rather than the creator. Ones that are cohesive and well made.

We’re in an exciting place right now – new IT products are changing the world in fundamental ways. It’s like the turbulent century after the invention of the printing press, the new rules aren’t clear yet.


What’s a market?

A market is the people who use a product. Or, to put it only slightly more indirectly, it is the route by which people come to know about and use a product. This is where the meaning of product and market blur.

Look at the Google search screen shot to the right (click to make it larger), for “complain about late train”. As you can see there is an advert “Bad train journey?” for the website FixMyTransport.

My question – is that Google search result part of the product or the market?

To a nerd who has just built FixMyTransport, it is clearly advertising. Part of the sales and marketing process, surely! In contrast, a general user on the Internet is barely aware that Google and FixMyTransport are separate things. They just know you type stuff into a box, it takes you to something that does what you need. For them, the Google search result is a vital part of the product – the first menu in the user interface for it.

So, for a specific product, a market is a specific set of people who use it. In its most extreme form it isn’t the list of people who could use it, but the list of people who do use it.

Are products and markets the same thing?

Like a hand and a glove, product and market mesh together. If you alter the product, its market automatically changes. You can design a product without thinking about its market, but to do so is merely ignorance. There is a market that you are making, you are just not thinking about what it is.

The marketing of a product is part of it. A Google search result is part of a user’s experience of FixMyStreet. The feel-good brand advertising of Coca Cola is vital to people enjoying its taste. Ubiquitous Apple stores both sell Macs and make it obvious where to get them fixed (“whole product”). The more you look at it, the harder it is to tell which is product and which is market.

It seems there is a line, just as there is a line between our brains and our skull. But it is a tightly fractal line, they fit together perfectly.

My analogies of gloves and skulls are a bit too glib. If you know about biology, a stronger analogy is perhaps one to genotypes and phenotypes. Our genes are analogous to the product. They express themselves, creating an organism. The product comes into existence, and naturally creates a market. Just as it is hard to predict what organism will come from the genes without running an experiment, it is hard to predict exactly what market a given product will create. You can research it a bit, and try and guess, but ultimately you have to do a minimal test to see.

That’s what all the lean startup stuff is about.

Geeks used to just be getting their technology to work at all. The usability revolution of the 1990s, which has now to a large extent played out, made them start to think about what they make as a product for ordinary people. Now, there is a marketing revolution of the 2000s. Geeks are starting to think about marketing as a vital part of what they make.

The obvious expression of that is the way the hackers at Facebook use the same skills that used to be used to write compilers, to instead optimise the growth of their market, and the fit of their product to it. Put like that it is slightly creepy… And yet, it is making stuff you like to use, right?

Products and markets, they’re the same thing. They’re MC Escher’s intertwined beasts. They’re a child whispering to his friend about the latest toy fad. They’re Tesco’s loyalty card optimising store locations.

They’re figure and ground. They’re musician and music.

Products. Markets. You’ll go better if you think of them as one.

Do epic shit! On being the first friend of DoESLiverpool

Today’s audio blog is about the excellent DoESLiverpool, and why you should be their friend too.

Being a friend (who does epic shit) of DoESLiverpool costs £9 a month.

So far the perks are, on the surface, imperceptibly different from those that anyone can get – excellent events, free hotdesking if you bring cake. You also seem to be able to bypass their otherwise authoritarian rules about lending out library books.

If you’d like to join, please contact DoESLiverpool. Ask them for their bank account number so you can set up a standing order.



On the need for a new profession

Complaining that I have a deluge of interesting things that I never get round to blogging, Neil suggested I use audio instead. Here’s the first one.

Links to the topics in this audio blog:

For a podcast feed, see my Audioboo profile.

Part 2: Why I think PledgeBank failed, when GroupOn and Kickstarter flew

This is the second part of a series of two posts. Part 1 explains how we came to make PledgeBank, what it does, its similarities to GroupOn and Kickstarter, and how we iterated on the product. It ends asking, “Why did GroupOn and Kickstarter succeed, when PledgeBank failed?” This part gives my answer.

Here’s what I think:

1. Giving up.

In his essay “How not do die“, Paul Graham points out that the real reason startups fail is that the founders give up. They stop changing the product, they stop doing deals.

We spent from June 2005 until probably mid-2007 trying lots of things out. But as this blog post in 2008 shows, by then it wasn’t something we were working on any more.

Tim Morley, who is an absolute hero, continues to run the site as a volunteer. And it continues to do amazing things, like help rebuild a burnt out furniture shop after a riot. But we stopped iterating on the core product –  removing things that didn’t work, trying new things that did.

Because of that there was no way it would go to the next level of usage. We gave up.

Why did we give up? Opportunity cost. mySociety runs lots of amazing sites, so there were other things that gave our limited time and energy a better (social!) return on investment.

But we have to accept that as a result, we missed making something as impactful as Kickstarter or GroupOn.

2. Being ideological.

We didn’t accept money. Even though some of our most successful pledges involved promises about money. Oh we talked about financial escrow really early on.  Looking in my email, about once a year we had long detailed threads discussing how hard it was to implement.

Sure, there were genuine difficulties with credit cards in the UK and with Paypal at the time, and we had fears about chargebacks. But we had overcame much greater difficulties to do things we cared about before that.

No, I think the technical excuses were just that, excuses. Really, we didn’t like the idea of accepting money. The original product idea wasn’t about money – it was about people doing things together. That’s what we were trying to make happen.

Theoretically though I now see that this is bunkum. Money is just too useful an intermediary of value that people already understand. As Kickstarter has shown, even for social goods.

3. Overestimating altruism.

We thought that the benefit of everyone else doing the thing, of the collective action completing, would be enough. It isn’t. Kickstarter would in theory work as just a pot you throw promises of money into.

But it does more than that, every project has a selfish thing you get just for promising. So, for example, if you fund this documentary as well as the benefit of getting to see the documentary, you’ll also get anything from a free DVD of it to a year’s supply of fairtrade bananas, according to how much you pledged.

I asked Yancey Strickle (Kickstarter founder) about this at Newsfoo a couple of months ago. He said that right from the start they’d realised they had to give rewards directly according to donation amount to “avoid donor fatigue”.

I can’t remember thinking of even adding this to PledgeBank, but suspect we were so focussed on the beautiful dream of pure opt-in collective action, it never occurred to us.

4. Being too broad.

It was always really hard to explain what PledgeBank was. It, ummm, lets you do things together, with a condition! People would be baffled, what kind of things?

It’s a difficult decision in a product to tweak how broad or narrow it is, and it depends on its market (products and markets are the same thing, a topic for a future blog post). But I think for the nascent collective action market, we were a bit too broad.

In theory, every project on Kickstarter could happen on PledgeBank. Look, for example, at the successful Nouveau graphics card driver pledge. But in practice, even if PledgeBank took money, it’d just be more confusing to use for Kickstarter-like projects.

Kickstarter has a community. It has a simple strapline “funding for creative projects”. Magically, in some cases, making something narrower can make it clearer and better, and so bigger than something broader.

5. Wanting it to autoscale.

We had a culture at mySociety that all our sites would basically run themselves, with not even one full time geek on each one, and certainly minimal ongoing administrative support. The power of code would make them work.

(In practice we now have one customer support member of staff and loads of volunteers running sites, but that’s another story)

This meant that we never considered doing custom copy and design for individual pledges. GroupOn pays skilled people to write its collective action purchase deals.

I’m not sure now whether Kickstarter used professional copy – some of the early projects had such good videos and text, I thought it had been curated by the core team. In my brief chat with Yancey he said otherwise, although I was left confused if they’d never curated them, or just didn’t do so now.

I think that with better wording and design and pictures and videos and presentation, some of the pledges would have been much more compelling and clear as products themselves.

Other reasons

The above are the things we didn’t try product-wise that might have helped. Some other notes:

We didn’t remove features. Rightly we kept trying stuff, like local search and PDF posters. But we just let features creep over time, rather than removing them when they weren’t any good. We weren’t lean, testing hypotheses and measuring what we did.

We never tried to hide the conditionality. Neither Kickstarter nor GroupOn throw it in your face, it is just kind of implied or inferred. Group action is an abstruse concept in the abstract, we gloried in it a lot. Instead, like our technology stack, we could have had the goal of hiding it.

Your reasons

On Twitter, Gavin Starks said it was timing, that we were not commercial, and not in the USA. He’s right, in that we were focussed on success in a different way to a commercial startup (see above). And the USA does dominate thought on the Internet. However, Omidyar paid us to have a PledgeBank marketing person in the US for a year, so I don’t think it was the limiting factor here. We never made it really fly even in the UK. Finally timing… I’m not so sure, I think our timing was fine, GroupOn was started while PledgeBank was going strong.

On the comments on my last post

sil, Paul and Andy capture quite succinctly much of what I’ve said in a more long winded way above. Basically I think they’re right. They also add to my point above that PledgeBank was too broad, by pointing out that expecting people to do things was always going to be a struggle. (The money issue is the obvious solution to that, but I hope not the only one).

Paul and Martin both say that PledgeBank hasn’t failed… I don’t agree. It hasn’t had the impact it could or should have done. It is better to admit that and try and learn lessons. Paul, yeah GroupOn does feel a bit sour now, but had we even stumbled on a fraction of it, the best democracy geeks in the world would be millionaires, and think what they might have done then…

Finally, Tony says that PledgeBank stayed the same after launch. It very much didn’t in terms of features, gaining everything from geocoding to Facebook. But Tony is right that in terms of core vision it did stay the same.

The future

There’s still lots of potential for PledgeBank. Barnet council have paid for a special custom version of it, which leads me to lots of ideas for collective action in just the Government/democracy space.

e.g. Think if you were still taxed, but you had to allocate how it was spent via a conditional spending Kickstarter-like site anyone could contribute ideas to. Or imagine a GroupOn for Government run services, that gave you daily discounts on skills training or swimming pools.

Finally, Martin and sil still like PledgeBank’s original vision. And I do too. Maybe, somewhere, somehow, someone, manages to make it fly. To quote sil:

Pledgebank wouldn’t be “an unsuccessful Kickstarter”, it’d be “easily the most successful coordination-problem solver”

Part 1: Why did PledgeBank fail when GroupOn and Kickstarter flew?

“I’ll do X but only if N other people will do Y”

I was abuzz.

A cafe in Holborn. 2003. A secret meeting. James Cronin and Tom Loosemore, who’d made the genius does-what-it-says-on-the-tin FaxYourMP, bought me coffee and changed my world.

Wow, there’s actually other people who think computers can revolutionise democracy!

And they’ve an amazing top secret, ludicrously ambitious project to repurpose the whole of Hansard to make Parliament easy for the masses.

A little later, Tom Steinberg launched mySociety with a brilliant call for proposals. You can still read the entries here. But despite my already high ambient level of excitement, the one that I loved the best was PledgeBank.

Go and read the original proposal now.

It now seems strange, but once we got the money and started building WriteToThem, all I wanted to do was get on and build PledgeBank.

Oh PledgeBank!

I’ve never been really left or right wing. And what is brilliant about PledgeBank (and indeed all mySociety sites) is that both people who love Government and people who love business, love it.

The reason I liked PledgeBank was, in economic terms, simple. Typically, we either act alone by voluntarily buying something selfish for ourselves, or we let Government use the threat of violence to take money from us and spend it for our collective benefit. The former lacks scale, the latter lacks an immediate link to needs.

But with PledgeBank, you agree to do something collective, but only if lots of other people will too! Best of both worlds. The thing you do is voluntary, and yet magically it also has the power of impact of masses of people doing it! Lovely.

I’m sure there were several .com bubble startups using collective action, but by 2003 none were very noticeable – there was nothing obvious on the Internet that used this amazing collective action idea.

It was obviously going to change the world.

And yet, it failed.

That might seem a bit harsh for something that created Britain’s preeminent digital rights NGO, that kicked off the first wave of invasion of New Hampshire by libertarians, and that even now has successful pledges every week, and is altering the relationship between city and citizen in Barnet.

But it isn’t worth $13 billion, like GroupOn, and it hasn’t had a million people backing projects, like Kickstarter.

At first it might seem like a bit too much chutzpah to even compare PledgeBank to GroupOn and Kickstarter.

It’s not though – long after PledgeBank launched, Andrew Mason (CEO of GroupOn) started out by making a very similar site called The Point. It tried to organise collective action consumer boycotts. The Point even tried to buy out (er., merge with) PledgeBank to kill the competition (we were too hippy to accept).

Eventually they tried GroupOn as a sub-experiment, and it flew. GroupOn is collective action – it started out basically as streamlined pledges of the form “I, a restaurant owner, will give a 50% discount but only if 20 people will come try out my restaurant”.

It’s also not like we tried nothing making PledgeBank. We iterated with unstoppable passion to try to get it to take off.

We had pledge signing by SMS, automatically generated PDF posters, translation into a dozen languages, pledges in dozens of countries, beautifully copy edited hassle emails, local geographic search alerts, cobranded sites for companies and charities, geocascading pledges, Facebook integration… All back in 2005-2007.

So, what do you think? What went wrong?

Why did GroupOn and Kickstarter succeed, when PledgeBank failed?

Your thoughts in the comments.

This is the first post in a two part series. Read why I think it failed in part 2.

Newsflash: Geeks now good at usability, everyone else crap

We really struggled.

We felt guilt. Wracked with pain.

It’s the mid 1990s, and computers are impossibly hard to use.

Anyone who could program them, and who also cared about people, was ashamed.

So we fixed it.

Books were publishedwebsites launched, a new profession was born.

It took a while but many of the key insights from that furore benefit us all every day.

For example, on an iPad, or in Google Docs, you don’t need to remember to press save to not lose your work.

It’s now impossible to start a new Internet company without its proposition being clear and explicable to the general reader, with a control flow that leads anyone through to a happy conclusion.

The most important revolution in the technology of information is now accessible to billions.

It’s every other industry that sucks.

Three examples…

Music – I live in Liverpool, which I suspect has an interesting, thriving music scene. But it is completely impenetrable.

Look at the flyer to the left (click for larger version). If you haven’t heard of Wretch 32 or Spank Rock, it tells you nothing.

What genre is the music? How varied is it? Might I like it? How much does it cost? What is that weird fuzzy blob in the top right, and the strange # sign? What’s an early bird?

If it was a computer startup it would be held to an even higher standard – you’d have to assume the reader didn’t even know what a “festival” was.

And this is one of the good examples.

Partly I think it is in cliques wanting to keep people out as they don’t have room in their venues. At least, that is how I feel as an outsider. Just like normal people felt about computers in the 1990s.

Every time I walk past a wall of such flyers, I gaze longingly and mystified. With no time, with no usability, I move on.

Another starving musician loses access to my discretionary spending.

House buying Diagrams such as this are complex, but actually make it look relatively straightforward.

The process, especially in the UK, is insane.

The incentives are distorted, customer service low, paperwork excessive. And that’s even when you’re paying a decent lawyer.

Let’s just say, Steve Jobs hasn’t had any influence over the design of the property purchase system.

There are obvious improvements Government could make, but they screwed them up.

Estate Agents are locked into old business models, and seemingly don’t care.

Sellers stubbornly refuse to drop prices to actually sell their property, as if the market was being fixed by an evil demon, rather than natural laws of supply and demand.

Buyers get the information they need at the wrong time, forcing them to unnecessarily renegotiate or drop out.

Mortgage companies require copies of byzantine sequences of documents, with no logic, sense or humanity behind it.

And heck, even the usable alternative of rental is unusable, as there are inadequate rights by social convention (no pets, no painting, no security of tenure…), and not even a decent system where the community of renters can praise good or shame bad landlords.

In short, it is crapper than even a 1990s computer by far.

Government – You can probably all think of a recent occasion when you found part of Government excessively hard to use!

Benefits and tax systems that are so complex, so time wasting you can’t optimise them, unless you are weak enough that your local council pays someone to work it all out for you, or strong enough you can afford tax havens.

Democratic systems so unresponsive, so unaccountable, voting not only seems pointless, but is pointless.

At mySociety, we were obsessed with usability from the early days. A significant part of its purpose is to spread “usability” to Government services, by involuntarily making them more usable.

Nowdays, parts of Government are valiantly trying to fix such problems, but even that has come from the computer geeks. At best though, it’ll be lipstick (really lovely lipstick! that will show the way! but still lipstick) on a pig.

Where are the people on the inside radically revamping services throughout their supply chain to be awesome? Fixing the “whole product” of Government.

I’m sure they’re there, but I’m also sure there aren’t enough of them.

(Next time all this annoys you, take positive action by slinging mySociety a donation, they’re taking action on it at all sorts of levels, not just in the UK but internationally these days too).

So yeah, enough guilt from us geeks.

When’s the rest of society going to step up, take responsibility for their parts, and make everything usable?

Astonishments, ten, in the history of version control

“If you really want to … truly ancient history, you have to go back to delta decks on punch cards.” (Jim Rootham)

In a world where biographies of cod are not just accepted, but rightly popular, it wouldn’t seem entirely crazy to write a history book on how computer programmers store the vital product of their labours – source code.

Since neither you nor I have time to read or write such a thing, we’re going to have to settle on this one blog post.

It’s an important subject.

The (for now) final end product seems incredibly obvious. And popular.

Yet it took decades of iterative innovation, from some of the cleverest minds in the field, to make something so apparently simple yet powerful.

And every step was astonishing.

1. Source code is text in a file! (1960s)

With hindsight, it’s obvious that source code is best stored as just writing in simple documents. A brief read of the history of ASCII gives a flavour for the complexity of agreeing even that.

2. Humans can manually keep track of versions of code! (1960s)

As everything, to begin with there was no software.

“At my first job, we had a Source Control department. When you had your code ready to go, you took your floppy disks to the nice ladies in Source Control, they would take your disks, duly update the library, and build the customer-ready product from the officially reposed source.” (Miles Duke)

3. You can keep lots of versions in one file! (1972, 1982)

Using a fancy interleaved weave file format, SCCS ruled the roost of version control for a decade.

It took some years to develop a good method for recording the changes from one version of a file to the next. “An Algorithm for Differential File Comparison” is a relatively late paper to read on the subject (1976).

In 1982, SCCS’s successor RCS (original paper describing it) used these diffs in reverse to beat SCCS, and astonished this commenter:

“Along came RCS with its reverse-deltas, and I thought it was the bee’s knees” (Anonymous)

4. You can each have your own copy checked out! (1982)

At the time, people tended to log into a central mainframe and work together via that. With RCS, using symbolic links, it could be arranged so that each person was working with the same version control, but their own working copy.

“there will be a file called RCS that is a symbolic link to the master RCS repository that you share with the rest of your group members” (Information on Using RCS at Yale)

5. Wow! You can version multiple files at once! (1986)

Amazingly, up until CVS, each version control system was for separate individual files. Yes, you can use RCS with wildcards to commit multiple files, or mark particular branches. But it isn’t really part of the system.

In CVS it was the default to modify all the files recursively. Software was suddenly a recursive tree of text files, rather than just a directory or an individual file.

It was badly implemented as it wasn’t “atomic” (successor Subversion fixed this in 2000), but really that doesn’t matter for the purpose of astonishment.

6. Two people can edit the same file at the same time, and it merges what they both did! (1986)

In the late 1990s I worked at Creature Labs. We were changing from Visual SourceSafe (commercial, made by Microsoft) to CVS (open source, made by a bunch of hippies).

There was frankly disbelief that it could do its main magical promise – let multiple people edit the same file at the same time, and be able to flawlessly merge their changes together without breaking anything.

The exclusive locking of SourceSafe was a real problem when we were making Creatures 3. I remember a particular occasion we were adding garbage collection which meant editing most code files, and the lead programmer had to check out every file exclusively over the weekend while he implemented it.

This paper from the 1986 is an excellent historical record of this magic, wherein Dick Grune suffers the same problem while his team code a compiler in Holland, and so invents CVS.

7. The shared repository can be on a remote machine! (1994)

Most of this time people were mainly using version control on one computer. Some versions of RCS, and hence CVS, had a remote file sharing mechanism to let you have a remote code repository in 1986.

“If a version of RCS is used that can access files on a remote machine, the repository and the users can all be on different machines” (Dick Grune)

But it looks like it was only in 1994 when a TCP/IP protocol added, that the idea really took off.

“[CVS] did not become really ubiquitous until after Jim Blandy and Karl Fogel (later two principals of the Subversion project) arranged the release of some patches developed at Cygnus Software by Jim Kingdon and others to make the CVS client software usable on the far end of a TCP/IP connection from the repository” (Eric Raymond)

8. Free open source version control hosting! (1999)

This isn’t an advance in source control technology, but it was astonishing, and on the Internet social advances can be as important as technical ones:

The tendency was for older OSS versions to be hard to find … John T. Hall had the insight that if projects were developed on the site, the old versions would be there by default. A development platform service was audacious, but no one else was doing it, and we thought “why not?” (Brian Biles)

Partying like there was no tomorrow (for their stock), VA Linux introduced SourceForge to the world. This was great for new projects (like my TortoiseCVS).

It was hard and expensive to get a server on the Internet back then, and it wasn’t easy or cheap to set up source control and a bug tracker. This new service, despite its lack of business model, fledged numerous projects that bit earlier.

9. You can distribute it all so there’s no central repository! (2005)

There was a wave of version control systems in the early noughties, making version control completely distributed.

That is, your local machine has an entire copy of the history of the code, and can easily branch and merge on a peer to peer basis with any other copy of it. By the way, the same feature makes it much easier to branch and merge in general.

Given that, it seems unfair that I’ve dated this astonishment 2005. That’s because I’m not recording the first time anyone made the astonishing thing, but the first time it was productised and became popular. April 2005 was when both Mercurial and Git were released.

The post “The Risks of Distributed Version Control” (late 2005) shows how radical this new-fangled stuff was seen to be.

10. When you checkout that’s a fork too, and you can do that in public! (2008)

The success of GitHub is for several reasons (that deserve a whole blog post, although I’ve alluded to one of them before).

In the context of this post, the astonishment was that you might want to make even your tiny hacks to other people’s code public. Before GitHub, we tended to keep those on our own computer.

Nowadays, it is so easy to make a fork, or even edit code directly in your browser, that potentially anyone can find even your least polished bug fixes immediately.


Have a quick look back up at those decades of progress. Yes, some of the advances were also enabled by increasing computer power. But mainly, they were simply made by people thinking of cleverer ways of collaborating.

It makes me wonder, what is next? What new astonishing thing will happen in version control?

More broadly, can the same thing happen in other fields?

Are core parts of our information infrastructure – that ultimately block innovation in government or healthcare or journalism or data, as capable of such dramatic improvement?

I have this feeling we’re going to find out.

Want more? Read “The version control timeline” (on Plastic SCM’s blog, don’t miss the comments) and “Understanding Version-Control Systems” (by Eric Raymond).

Why I just joined the Green party

I’m pretty sure lots of people are going to ask me why I just joined the Green party, so here’s the reasons.

  1. I’ve moved to an area of Liverpool where green is strong – two Green councillors live nearby, there’s a very local organic box delivery, and the main indy co-op food shop has just moved up the road. It’s nicer to be in a party when there is local activity to join in and build on.
  2. Since I’m no longer being paid to work for mySociety, I’m no longer worried about any accidental partisanship being a member of a party might bring. Not that I think you can’t be a good civil servant or work for mySociety if you are a member of a political party, but it requires extra energy. Lack of bias is extremely important for both.
  3. I’m quite worried about Climate Change (and other things). We need to continue to try to both prevent it and to build up our societal resilience to deal with the consequences of it. We can’t do either unless we act in groups, and politics is part of that. More on this another time.
  4. Strategically I think the Greens are where it’s at in Liverpool. Democratically, i.e. to meet the needs and wishes of Liverpudlians, what Liverpool needs is a (slightly) more “left” party than modern Labour. Every city needs a strong opposition, and the Greens are the best candidate.

Finally, and to be clear, I think the Labour party, Liberal Democrat party and Conservative party are all excellent. I think the recent Labour Government did many good things particularly in its first term, and I think that the Tory/Lib Dem coalition is doing many good things and will continue to do so too.

I don’t join a party because I agree with everything it thinks. Fundamentally, of course I don’t, I’m unique! And I’m not going to lie and pretend otherwise. Every party is internally a coalition. Call me on it if I ever drink the Kool-Aid and start claiming it is more than that.

In this context, one good thing about the Green Party is that it is democratic. You can change policies using a documented, democratic process. For example, their science policies are in the middle of much needed reform.

Lots of work to do, and that’s the whole point. You don’t get input into the political system by not putting in any input.

Parties have had a bad reputation in recent years. There’s been a kind of twisted war with modern campaigning and the media ratcheting the cynicism up in a spiral – until you can’t work out if it’s voters or politicians who are the worst in their systemically forced negative actions.

I’m having no truck with that, and nor should you.