History of libraries

Thanks to the excellent Thinking Liverpool [1], a lunchtime talk about the history of libraries caught my eye.

I mention the history of public libraries worryingly often. I make an analogy between libraries in the age of the printing pres and the modern need for equivalently novel public institutions in the age of the networked computer. Time to find out more.

It was in the Victoria Gallery, just across the road from work. Great!

Victoria Gallery and Museum

Mark Towsey, the historian speaker, said many things. I’m going to highlight a few at random which struck me.

The price of books has shrunk in imaginative ways for centuries. One unusual example of how is in the form of James Lackington, a bookseller in London from the 1770s. He noticed that when sales of a book slowed, publishers kept the price full and pulped anything they couldn’t sell. He saved these books from destruction, sold these “remaindered books” at knock down prices, making both lots of money and increasing use of books in society.

This is a picture of the “The Temple of the Muses” bookseller which he started in Finsbury Square.

Temple of the muses

Liverpool had several interesting early libraries. In 1758, The Liverpool Library was established in a schoolmaster’s house. By 1802 they built the Lyceum building at the bottom of Bold Street to hold the library. It’s now a somewhat destitute building – Mark said that if anyone has a bequest of a million pounds, he has a plan to turn it into a beautiful museum of the history of libraries!

The Lyceum

The Liverpool Medical Library was started in 1799 by a group of important Liverpool doctors. There were some very expensive medical texts at the time, that local booksellers didn’t stock. One doctor could only afford a small number of books – together lots of doctors could buy them all. (Reminds me of what PledgeBank was trying to achieve with our local pledges).

This is Robert Hooper’s “A Compendious Medical Dictionary, containing an Explanation of the Terms in Anatomy, Physiology, Surgery”, published in 1798, and perhaps the kind of book they were buying. (I’ll try and get back to the Gallery to take a photo of a real book they actually bought, unless anyone already knows of one online…)

Hooper Vade Mecum

As caricatured in this painting by Isaac Cruikshank, circulating libraries concentrated on more downmarket books. At least, the romances and the novels in the cartoon are all taken out, whereas the worthy histories and philosophies are still on the shelf. Not so different from complaints in my lifetime about television being downmarket – just before it entered its current phrase of Dickens-like quality.


The Circulating LIbrary

At each stage of this, books became cheaper and more accessible to more people, and literacy levels rose. One of the final steps in free access to books for everyone was the Public Libraries Act 1850. That’s a full four centuries after the printing press seriously came to Europe.

Afterwards, I asked Mark about the analogies I’ve been making between the modern world and libraries. He wasn’t aware of anyone who was specifically researching or thinking on that basis. A “future of libraries” meeting a few years ago was about the physical spaces, not, say, about public space on the web.

He did say that there are numerous ideas to mine from the history, some which worked for a while, some which didn’t work, as well as those we still know and use now. It turns out the vast, wild sea of ideas of the modern web, things like Patreon and Wikipedia, is an echo in new technology of a soup of ideas a few centuries ago.

He recommended that I should read A Nation of Readers by David Allan. Alas, a bit out of print!

A Nation of Readers

The talk was in honour of a new exhibition a the Victoria Gallery, of very old Liverpool books. Hopefully I’ll remember to go back one lunchtime while it is still on.

[1] A site which lists interesting talks in Liverpool each week. Interestingly, scraping wasn’t good enough to make the site – Paul instead outsources some of the process of finding a curated, clean list of events via UpWork to a fully qualified, fluent English speaking lawyer in the Philippines.

Don’t play the game, mutate the board

I found some notes I’d written at the start of this notebook, a few months old.

Notebook cover

I used to campaign in Cambridge for more Fairtrade coffee and chocolate. That was a “closed” tactic, as opposed to an “open” tactic like open source software or open data.

It worked – even the best selling chocolate bar Dairy Milk is Fairtrade now. The tactic slightly shifted the board of global economics.

These notes are about this question – what tactics create the change in the world that I want?

Fairtrade was a closed tactic of certification (for many!) which I loved and which shifted the board

Simon Wardley has a kind of map of evolution of technology. He encourages organisations to make such maps to improve their strategic awareness.

Vinay Gupta made a game called nuclear poker. It has simple rules that simulate the important game theory of nation states and nuclear weapons.

I played it once, and to start with I wanted to win! The natural urge, when just told the rules, is to collect the cards, to gain abilities. Before I knew it I was thinking I might nuke someone pre-emptively to stop them becoming a  superpower before me… Then I realised I’d turned into a psychopath.

(I role played the next round as Hans Blix, weapons inspector, trading and discarding key parts of the nuclear tech stack to eliminate them from the game)

Much like Vinay’s poker game, when I see Simon’s maps of tech, they at first make me want to win. Unfortunately, because of the structure of capitalism and technology, that essentially means frig the game so you get a monopoly and make billions.

However, the idea doesn’t inspire me. I know the monopoly would have to be broken down later to make things fair. And that it’ll happen anyway, someone else will find it. It doesn’t feel like it changes the game.

(Of course I’d love to make billions, and partly I’m just hiding because it is hard, and risky to try more. But nevertheless, as a goal it doesn’t inspire me like it does some people.)

Simon's board makes me want to play chess... (like Vinay's nuclear game) ... to win! Which implicitly in capitalism means make a monpoly and make money in $Bn.

I’d much rather mutate the board, change the game. Or force others to mutate the board for me.

In the past I’ve used tactics to do that – more open, more innovative, more fun! They’re not directly what I’m after, they’re just new things to use that let me change the board.

Actually what I want is 1) to end poverty of all kinds, 2) to make civilisation fundamentally sustainable for tens of thousands of years, 3) to fully understand the universe and our minds.

For me winning is mutating the board and forcing others to mutate the board to make it more open, more innovative, more fun, to end poverty, sustainble, understand universe. These three crosed out are tactics or actions that worked for me before. They are not goals.

My dumb error for a while was to think it was the framed societal goal (money) of the board, or the tricks that worked for me before (open) that I liked.

My dumb error was to think it was the framed societal goal (money) of the board, or the tricks that worked for me before (open) that I liked.

I was around at the start of both Open Knowledge and mySociety. I always preferred mySociety’s tactics, because they were about an end goal (usable Government) rather than about a means (open data).

You can blindly promote Freedom of Information (it’s under threat by the way, and you can help) in the hope that it reduces corruption. It might not. You can promote open data in the hope that it improves Government efficiency. It might not.

In contrast, the aspects of mySociety that were about usability – they were more straightforwardly successful. The means were also the end. And they inspired the Government Digital Service to take that further.

What we were doing at mySociety was taking the culture of open source and the culture of the web, and applying it to Government. We didn’t strategically know what we were doing. Although we had a strong sense that this new technology could help with some political problems.

That is OKFN's error against mySociety. FOI may be (or may not be - leaks better!) a good tactic against corruption and inefficiency. Random open data may help with efficiency or fun. Showing user centric government digital as possible did create GDS in the end. We were playing with open, usable web culture to see what good new things it could produce. Largely blind to the strategic situation.

The next, and final, page of my notes gets less clear and more speculative. Here I think I’m saying that the overall strategy of large, classic NGOs isn’t very clear.

It is those NGOs which are trying to deliver the end of poverty, the sustainability which I gave as goals earlier.

The NGOs have no strategic play either.

I’m now wondering, where can the strategy for those things come from? How can we learn to change the game so the whole system is more sustainable, or has less poverty?

What created the web and open source culture, that our intervention at mySociety completely depended upon? I’m curious how to make more such cultures.

Is the Wardley map any good for this kind of analysis? I suspect not – it is designed for technology capitalism. Which is perhaps why I always find it frustrating. Maybe there is a way of using it that will help with what I need.

Either way, Wardley’s innovation/commoditisation cycle with dependencies and user needs is useful background.

Why was web culture there??? Wardley maps any use! or too focussed on tech capitalism? Nothing on risk, say. It is not the map that is clever, but understanding techs innovation/commoditisation cycle and its dependency on other tech and user needs. Innovators dillemma says new things that beat commodity will come from bespoke.

I leave the question hanging, how do we map the board so we can make interventions to improve the overall outcome of the game?

Back page of notebook

Sync/Backup workshop at Redecentralize Conference

The fabulous Redecentralize Conference was organised by Ira and a bunch of other volunteers. Its subject – how do we make the net resilient, private and fun again?

It was an unconference, so I decided to do a session on a personal itch I’ve had for the last few years – file synchronisation and backup.

I don’t have a nice way to manage my files – documents, music, photos, email. Syncing them across devices and keeping them backed up seems strangely harder than it did in the 1990s. At least, if you don’t just go all in and trust Google or Dropbox with them.

Redecentralize Conference
Photo by Luigi Bozzo

The idea of the workshop was to find out what decentralized software technical people actually use every day for syncing and backing up their files.

We wrote down all the tools on cards, and voted with sticky notes if we used them. I was hoping there was some amazing tool based on newish technology like Distributed Hash Tables that I could use. There wasn’t.

So here’s the tools geeks use, working up to the favourite at the end.

Syncthing (0 votes)

This is one of the few actually peer to peer tools which came up – it directly syncs files between devices. Lots of people mentioned it, which is why it got a card even with no votes. Alas, none of us use it day to day. It just isn’t mature enough. Every six months I try it again, hit an unsolvable problem and revert to BitTorrent Sync. They’re working hard to improve Syncthing, and I’m sure would welcome more help.

PhotoBackup (1 vote)

Looks like a nice solution to get files off your camera phone and onto your own server. It’s a pain that it needs a special server to post the files to – I’d like it more if I could configure it to just use SCP (see the SSH section below).

Carbon copy cloner (1 vote), Time Machine (2 votes)

A few people recommended proprietary Mac tools. They considered them decentralized, because they keep the backups on their own multi-terabyte hard drives at home, rather than putting them in one central cloud. No doubt they’re polished and easy to use – I still use CrashPlan on an old Mac for that reason! Alas, they’re no good if you have even one device with another operating system.

ownCloud (2 votes)

This self-hosted email/calendar/contacts/file sync tool is a powerful combination. Surprisingly few people use it. My theory is that it tries to do too much itself, and does none of it quite well enough. I found it too slow, clunky, and unconvincing to trust my most valuable data to. I’m optimistic that it’ll continue to improve.

duplicity (2 votes)

While researching backup for myself, I find price is a constant frustration. All the most open ones require a server with a live disk mounted on it. S3, an Amazon service that can just store blobs, is considerably cheaper. It’s even cheaper than Dropbox, which is built on top of S3 and adds a margin!

A couple of people at the workshop took advantage of this price while keeping privacy by using duplicity. It encrypts files and stores them in S3. I find that a bit of a cheat – some control is still lost, it isn’t resilient. As economies of cloud storage scale even more, I’m open to changing my mind!

Tarsnap (2 votes)

This technically excellent Unix backup tool also uses S3 behind the scenes. It is by all accounts easy to set up, fast and powerful. My one concern is that it is too dependent on its founder Colin Percival. You really want backup software to have a whole community maintaining it, or a strong company with incentives to keep the service running. Or it won’t be there when you really need it.

SSH (4 votes)

People using SSH probably overlap with the rsync people below, or they were using scp or bup or unison or something similar. All of these excellent command line sync and backup tools need a protocol to send files remotely, and that’s SSH. Stalwart, and to this day what geeks use to write data remotely from automated processes. It’s that basic writing functionality which I want some funky new DHT to improve.

rsnapshot (5 votes), rsync (1 vote)

It’s hard now to remember how revolutionary rsync was when it came out (apparently in 1996!). Rolling checksums make file copying fast the second time, enabling all sorts of new backup and sync possibilites. The raw technology idea was vital to Dropbox’s genesis. Nearly everyone at the workshop used rsync via rsnapshot, a layer which can backup whole machines and archive their history.

git (6 votes), git-annex (1 vote)

I use git with an auto-commit script to keep important text documents. One workshop member used git-annex, making even large files not a problem. It’s very geeky, but it’s very reliable – history and checksums and decentralization make it very hard to lose data. It does two way sync, but unlike Dropbox actually merges the files.

A key problem for me with this is that Android support is woeful. In the end, I fix that with a nasty hack – I use BitTorrent Sync to transfer a directory on my phone to a server, and have a script there which merges and commits.

Thinking about it, git is the biggest obvious opportunity which came from this workshop. Just as Dropbox was built on rsync technology, could something very user friendly be built on top of git, for the purpose of personal document syncing? Add a peer-to-peer transfer protocol, a beautiful GUI, three way merge of Word documents… The internals built by Linus would make it robust and reliable.

Why you might want to decentralize your files
Why you might want to decentralize your files

OS file browser + standard protocols (1 vote)

At the end I asked the room, “who is happy with their solution?”. Only one person put their hand up. She had tried every nerdy backup tool imaginable, and abandoned them all.

Instead, she keeps files on her computer and her relative’s computer. If she’s making a document she cares about, she transfers it by USB stick between the computers, to make sure there are multiple copies in different places. And that’s it.

Unsophisticated? Not at all. Way easier than backing up paper. Understandable by anyone. Full control. Cheap.


Thanks to everyone who came to the workshop. I was disappointed that there wasn’t a solution to my problem. However, I learnt several valuable things, including one to improve what I do in the short term.

I need my own (virtual) server (again) – there were several folk from Bytemark at Redecentralize. They made me realise that I had been trying for a while to not need my own server. Alas, the fully distributed solutions just aren’t good enough yet. You need something you can make web requests to and SSH to if you want to use OwnCloud, git, ssh or rsync. It’s a relief to accept this.

Simple is better, sometimes – the happiest person was doing manual file copying like you would have done before “the cloud” in the 1990s. It’s obvious to non geeks, don’t be afraid of doing this. Of course, you accept downsides, but you gain upsides of simplicity and peace of mind.

Someone will win with an inspiring UI on top of Git – merging of text files, robust history, fast and reliable. There are fundamental technical reasons why git is good for some of this stuff. Just as Dropbox used rsync technology and brought it to hundreds of millions, one day somebody will bring these benefits of git to the world.

We need low level distributed storage / naming – personal servers running Dropbox clones will keep us going for a while, but at the end of the day this worrying about servers, about named computers, is part of the problem. At some point, a distributed file storage will take off. IPFS is exciting, although it has no charging mechanic so can never be “fire and forget”. My best hope right now is MaidSafe. When they ship!

There is a market for a geeky command line sync/backup tool – despite all the open alternatives, nothing works well enough that it has dominated the market. I’m confident that technical people haven’t solved this problem for themselves, and want it solved. Powerful interfaces can then be built on that stable base.

I’d love to hear your thoughts and ideas in the comments below!

Making our information society safe and fair

The topic of how to make our information society safe and fair regularly comes up in conversations.

I think we need some quite big, radical things. They’ll need new public service Internet organisations to implement.

This is my high level view list.

1. Access to culture

“People have too much knowledge already: it was much easier to manage them twenty years ago; the more education people get the more difficult they are to manage.” (one MP’s response to the Public Libraries Act 1850)

Liverpool Central Library
The printing press wasn’t a fair innovation until centuries later, when Victorians made the public library widespread. Some of the Internet is free at the point of use, due to advertising and due to the free culture movement. Some of it isn’t.

Much of the most detailed, intelligent reporting – such as the Financial Times and the Economist – is not free. Much vital culture – such as the Wire – is not free.

There are numerous business publications and research organisations, which aren’t free.

I think we’ll need a method, similar in purpose to public libraries, which gives poor teenagers access to the same culture as rich ones. Which gives someone with few resources starting a new business access to the same data as a rich corporation.

Where to start: Build on existing physical libraries – make sure they have subscriptions to paid for aspects of research and culture. Build on free culture. Imaginative ideas for new funding methods.

2. Digital literacy

Peasants on a collective farm in Livezeni village, Arge? County being read a newspaper in 1950

Many technology innovations only reach their best when complemented with mass education.

We teach nearly everyone in the UK to read and write. A majority learn to drive. Without that quite complex training, the printing press and the car would be just for an elite.

What could everyone learn which would help us make better use of computers? Usability, which I love, will only get us so far. Until we have strong Artificial Intelligence (it’ll be a while), many people in society will need to understand computers well.

There are still plenty of technology changes to come, and we can’t know exactly what to teach until after they’ve happened. Meanwhile, we can make a start, and iterate as we learn more.

Where to start: Teach children to code. Make sure employees can use spreadsheets in a sophisticated way. Even teach police to type! (Richard Pope‘s idea after seeing a desk sergeant who was very slow at entering his paperwork)

3. Professional programming

Tacoma Narrows bridge

Engineers don’t build bridges that fall down, not any more. Software is constantly broken, our data stolen and our privacy breached.

There’s a whole lot of it which needs rebuilding in new ways which are barely researched yet. The industry needs to be professional to do this.

Ethical policies which help defend privacy. Quality policies which make it secure. Diversity policies which makes it usable for everyone… There are lots of things a professional programming organisation could improve.

I’ve benefited a lot from the accessibility of programming – I learnt as a hobbyist child from my father. We can make programming both accessible, and professional.

Where to start: Look at other engineers. Look at other professions, like doctors and lawyers. Join and improve ethical professional bodies. Consciously try to not harm the freedom which general access to programming gives in the process. Create standards.

4. Building regulations

Building regulations

Increasing skills is important, but won’t be thorough enough. We need to enforce standards by law too.

Some of these will be mundane but vital, such as websites using only a few standard sets of terms and conditions. Others will be life saving, such as making sure your driverless car manufacturer has a high standard of software engineering practice.

Where to start: Regulate to stop coding in unsafe languages. Support organisations like I Am The Cavalry (automotive software safety). Build on industry guidelines until they are mandatory (e.g. MISRA).

5. Ethical cryptography

First woman jury, Los Angeles

It would be foolish to use no encryption, allowing Governments and criminals to spy on everything we do. And completely unethical.

It is just as wrong to hope that all things will be encrypted in a libertarian utopia. There are criminals and enemies who courts should be able to get evidence from.

This blog post by Vinay Gupta describes the three actors which cryptography should model – the users, the Government and criminals. It describes a much more sophisticated threat model than we tend to talk about.

Where to start: Implement ideas like Cheap ID and jury-based crypto. Develop capability to practice a hybrid of law and tech. Use that to develop technical / legal systems similar to constitutions.

6. Power framework

Hal, 2001

There’s a war over who has access to the core ability of computers – to program them in arbitrary ways.

How is what computers do controlled?

To understand the issue, read Cory Doctorow’s article The Coming Civil War over General Purpose Computing. It’s a very complex question.

Where to start: Build on organisations like the Free Software Foundation, Electronic Frontier Foundation and Open Rights Group. Academic research which combines the abstract philosophy of delegated agency with practical user needs.


What do you think needs doing to make our new information society safe and fair? Leave a comment!

Those brief moments when winning seems possible

This mad dash bad mixen fun up down world.

Forces smash hither and fither, 7 billion of us strange qualia,
doing, being human.

Web culture, open source culture… sucked out of academia,
hacking, sharing, making, funning, building. Just late enough
for usability to be just there just cheap enough.

Smashed with.

Political culture… burnt out, tired, so so so good best
world has ever been, so so so broken, media corporate
capitalist socialist thought tangle.

Wonks, meet Geeks.

Geeky wonky wonky geeky geeky wonky geeky wonky geek geek
geek geek wonk wonk wonk wonk.

Only in England. Honourable neutral, civic servile, playful,
statirical. A soup, aglutinating not once but many times –
digitrights faxing, illegal-war voting, political surveying…

Pulled together into an organ, into a community, into a charity.
Inspiring. Beast poking.









Love us hate us join us be us. Pester your politicians, repair
your civic infrastructure, free your information!

there are few of those very brief moments those times when two
things combine in the right place with the right skills and
set the pulse and light up and burn and grow and all the
people and all the amazing people and spread spread from
this island round the world ideas filtering through a million minds
and alter the framing change the expectations uplift uplift
hope hope hope.

Those very brief moments.

When winning seems possible.

“I find politics inaccessible”: User testing voter advice apps

How is she going to vote? My friend (let’s call her F) just doesn’t know.

This isn’t unusual – about 20% of voters are like her (see Q3).

She needs to work it out fast. She doesn’t like polling stations, so has a postal vote. She has to decide this weekend – or she’ll get busy with work, and not remember to post the ballot in time.


mySociety helpfully list 16 voter advice apps, so I decided this was a chance to test them out!

This is a real voter, who doesn’t like politics – or to be precise finds it “inaccessible”. Exactly the kind of person these apps are trying to help. She’s a 30 year old professional, lives in Liverpool.

It’s a pretty harsh bit of user testing, I’m not going to sweeten my report of her reactions to the sites. So here goes… (Skip to the end for a summary if you get bored)

1. Who Should You Vote For? 0/10

We’re on mobile (my Fairphone 1, using Opera browser) – for one thing, we’re on a train and then in a coffee shop. For another, F says she would never do this on her laptop. If she had spare time at home, she’d be more likely to watch Netflix. Voting intention is relegated to spare time while waiting for a bus.

She read the first two questions of Who Should You Vote For? – one about a 50% tax rate for top earners, another about the mansion tax.

Who Should You Vote For questionsThen declared it boring. She doesn’t care about the questions, and there are too many of them.

2. Vote for policies 1/10

F immediately liked being offered a choice of issues that she cared about. She hated the wording on the more detailed questions that followed – didn’t understand them.

Vote for policies

She gave up after the first party. If she could have skipped to the second party she would have, but you can’t click on the numbers to do that.

My view is that the site uses wordings from manifestos, and these just aren’t written in an accessible way.

3. Position dial 0/10

“Pictures? What’s all this stuff? I don’t know what I’m supposed to do.”

Position Dial confusing

She closed the window. The site is just too confusing a mush, without a clear call to action.

4. iSideWith 9/10

F likes the questions. The interface on mobile was “a bit jerky” – for some reason it kept zooming in when choosing answers. There was also no explicit skip button for questions, but she found out that you can just not answer them.

She liked the “more stances” button, although it rarely satisfied her. In one case she actually typed in a new detailed one!


We lost her answers due to the mobile browser and the train. She didn’t mind – reentered them in the coffee shop. She liked it that much.

She got her answer: Lib Dem 98%, Labour 98%, Green 93%, Conservative 65%. She then tried the function to compare results, she wanted to know why the scores varied. What were the important policies? It wasn’t clear.

All told, the high score was because she found the questions comprehensible, of the right volume, and fun to answer.

5. Vote Match 7/10

It feels like a speed dating site!

“Too hard” – dragging the order of topics. F put the first three in order, but then got stuck. She didn’t know what “political power” meant or what “trending issues” meant. Nevertheless, she went to the next stage no problem.

Her first reaction is that she doesn’t like the questions much, but isn’t sure why. The iSideWith ones were just easier to understand. For example – the “find out more” button for the fracking question explains what fracking was, but gives no context as to why it might be good or bad, or what current policy is.

Vote Match

“The Government’s top priority should be too sort out the deficit” – F didn’t like the wording of the question. It is important, but surely top priority should be to keep us safe and not starving! (My view is that this is over use of standard political wording, which is alienating)

“Public spending should be maintained at least at current levels” – F doesn’t understand what “public spending” means – it doesn’t say what spending! Depends what they cut. Isn’t very clear. (Again this is a political class phrasing and argument, meaningless in day to day life)

“UK is better off in the EU” – doesn’t tell F the postivies/negatives. There is no find out more. Wants a list of advantages / disadvantages of being in the EU.

Result: Green 73%, Labour 70%, Lib Dem 63%, UKIP 38%, Conservative 30%.

F’s assessment so far: This is not helping – one quiz said vote Green the other vote Labour! They didn’t say why in either case.

(Later F tried the Telegraph version of this and quickly realised it was the same. She was skeptical, worried Telegraph would be bias.)

6. YourCandidates.org.uk 1/10

I’ve put in my postcode. It’s telling me what people are. Why? Tells me how to contact them. Hasn’t told me anything about them.

F didn’t see link to “Personal homepage”, she thought it was another contact link.


With prompting, she then tried a website of Green candidate. She found there was too much to read, it was irrelevant blog posts, nothing about policies. Wanted policies.

Neither she nor I noticed the “Choose comparison” dropdown, which lets you compare policies, until I was making this blog post later. It’s a bit clearer on desktop.

7. Tickbox 2/10

Weird that they all want my postcode.

The initial “Have I decided who to vote for” annoyed her – of course not, that’s why she’s using the site!

She’d heard people talking about Trident, knew it was about defense, but didn’t know what it was. She liked the explanation on the site – that it was about
cost vs. security. She likes the explicit “don’t know” button. “GP surgeries 7 days a week” – F was surprised! A new question that none of the other sites had asked.


Result: 1) Tory 2) Lib Dem 3) Labour

She didn’t like the answer, doesn’t believe she’s Tory (doesn’t like them, and other quizes didn’t suggest them). Also it only answered 5 questions. So she goes for 5 more questions.

10 questions in: 1) Lib Dem 2) Labour 3) Tory

No Green! Only 3 parties. Not useful. She likes the explanations of the questions on Tickbox, but just doesn’t believe their results. She kept going to see if they stabilised. Grammar schools – likes them, first time asked! (Clever kids get left behind in current schools, not fair)

15 questions in: 1) Labour 2) Tory 3) Lib Dem

Still doesn’t believe it. Also says that the “5 more questions” button isn’t very forceful, and loads of people would miss it. So get incorrect answers from too few questions.

20 questions in: 1) Tory 2) Labour 3) Lib Dem

“It is just making it up”. Likes questions and explanations, but pointless if it is just giving a rubbish result. She asks a general question – how can I know if these quizzes are bias?

8. Who Shall I Vote For 5/10

Too much text on the first page saying it isn’t bias, F didn’t read it and it annoyed her (!).

She likes the choice of issues again. Although now is worried might miss important things. She realises how hard it is to ask about the right issues in these quizzes. She didn’t want to pick 3 topics, it made her pick 3. Now has to do 10 questions on things she doesn’t care about.

UX pain – couldn’t go back and set scale on first question, when on second question. She doesn’t like the lack of any explanation of the questions.

Yay, first to ask about badgers! Likes that.


There’s then a principles section which asks really general philosophical questions about approach to life and decisions. She finds them really hard to answer, and has to skip loads of them.

Principles section

Male or female – why’s it need to know! (She skimmed and didn’t see the “research purposes” bit). She then refuses to tell it who she would vote for – thinks it may bias it, and that it is none of its business. I try to get clarity on this, but couldn’t – it was very instinctive reaction. Doesn’t want to give postcode, hated that, no reason for site to ask.

Result: 1) Green 2) Labour 3) SNP 4) Plaid 5) Lib Dem

Asked very detailed about 3 things she cared about. She likes breakdown of every question by party.  Overall though, a bit too detailed. Rather be asked more generic questions about fewer topics.

9. WhoGetsMyVoteUK 2/10

Sex/age/education – she hates being asked. The copy says “this app does not collect any personal identifying information”, so she says “why does it need this personal info then”? Skip!


She pressed wrong answer for a question, then the back button goes back to beginning not to the previous question. Rubbish!

Questions seemed OK, but hated losing all her work when trying to go back. Gave up quickly.

She’s getting bored :)

10. Verto 0/10

F doesn’t like being asked where born and where lives. Doesn’t want to answer loads of data collection questions.

This app has left/right swiping like Tinder, which she has never used. She can’t remember which means like and which means dislike! Noticing she’s being slow, the site pops up a helpful dialog.

Screen Shot 2015-04-25 at 19.58.24

That doesn’t clearly explain which direction is which, so she still doesn’t know! Gives up.

Overall view – she likes idea of playing a game, but it took to long to get there with other questions, and then had a confusing swipe interface.

11. Voting Counts Policy Matrix 0/10

F thought could click on Issues like NHS but can’t. Why such big space when not links? She complains she can’t compare breakdown of parties, that it should be in table!

It’s then that we realise the site doesn’t work right on mobile. I show her the desktop version on my laptop, and she likes the look of it – but that isn’t relevant to this test, which is mobile only.

12. Your Democracy 0/10

Got to create an account – not doing that. Can’t be bothered.

Your Democracy

13. Awedience -1/10

This site asked for F’s postcode, when she entered it it said “Unknown analysis”.


She was so annoyed that she’d gone to the effort of giving her postcode, but got nothing back, she gave it a negative score.

14. Election Compass 0/10

There’s a bug on mobile – can only see half the page, and can’t zoom or drag sideways.

15. The Economist 0/10

F at first worries they’re bias. Suspects they’re not, but isn’t sure why she thinks that.

She complains she can’t read it properly – fonts too small on mobile. Then a box pops up, and she can’t read all the content.


She gives up.

16. YourNextMP.com 0/10

Another site that just tells me about the candidates. Doesn’t help me decide who I’m going to vote for.


F felt it would be more useful a site to someone who is into politics and already knows stuff. She also felt she’d already seen it – same kind of experience as YourCandidates. Overall, boring.

17. Democracy Club CVs 0/10

I decide to test out this site, which I made! F knows it’s mine, but doesn’t hold off on criticism…

She enters her postcode, then says that because not everyone has submitted their CV, it is not representative. It does makes her think they can’t be bothered, so she sees that as a negative.

Not submitted their CV

It doesn’t help her decide, tells her about the person. She wants to know about policies.

I don’t want to vote for XXX because she has good experience, if she also kills badgers!


  • Do user testing. Lots and lots and lots of it.
  • Write the content properly. Don’t use wordings from manifestos, or any wonkish part of the political system.
  • Explain what things mean and current policy. Help the voter relate issues to their own knowledge / opinions.
  • Develop and test on mobile. May as well write it in Swahili if you don’t.
  • Several quizzes were rejected entirely because of small (in code!) UX bugs which made them unusable.
  • Don’t ask personal questions like sex / age / postcode, or even how voted last time. Off putting.
  • When giving results, explain simply why – what policies most distinguish the parties in context of what voter cares about.

The winning app is iSideWith! Second and third place go to Vote Match and Who Shall I Vote For.

Nobody else came close, all scoring 2 or less out of 10.

After all that, F still doesn’t know how to vote. She needs help choosing a criteria to distinguish Labour and Green.

Why I’m collecting every MP candidate’s CV

My side project for the last month is to try and collect the Curriculum Vitae of everyone standing for Parliament.

It’s called Democracy Club CVs.

I’ve been working every spare hour – mainly around midnight and on Sundays. Partly it is technically interesting, partly the other Democracy Club volunteers are fun to hang out with…

It’s also just enjoyable, with leaflets from ex-whips, professional musicians and raving loony leaders (that last one is standing against Boris).

But that doesn’t really explain … why am I doing it?

Background matters

The idea for the site came from this election leaflet Julian found in the 2010 election (Guardian article). It has a kind of CV written by Jacob Rees-Mogg’s opponent. Take a look.

Rees-Mogg CV Election Leaflet

I don’t have a fixed idea of what background makes a good MP. I do, however, like the idea (thanks James) that their background should be representative of their constituents.

Hopefully, when we have enough CVs, academics linked to Democracy Club can start doing this kind of analysis.

Layout and form

Aine, one of the directors at ScraperWiki, really likes CVs. It’s not that they’re magical things that tell you everything you need to know about a job applicant. Why is it?

Firstly, the form of the CV tells you quite a bit.

Is it clearly laid out, easy to read, spelt correctly? Is it evidently a CV they’ve used to apply for other jobs recently, or have they never had to apply for a job and it is just their political bio recanned?

It’s not just me that thinks this matters – we ran some user testing with unconnected members of the public, and it was the main thing they judged the CVs on.

To try and encourage good formats, we added this to the upload page.

This is a CV

In later desperation, I also gave a link to the National Careers Service advice on writing a CV.

Skills and talents

Secondly, the content gives some indication of skills.

Would our Prime Ministers like a better talent pool to pick ministers from? Would it increase quality in the way the country is run? I don’t know!

But it wouldn’t hurt to try!

You can pick a maths teacher,  a mental capacity lawyer, a manufacturing engineer, an expert in changing complex organisations, a carbon capture lead… And that’s just from the few CVs we have in already.

With a CV the details tell you lots – the exact list of places and names of companies, whether there are gaps. The progression in job titles. It’s all that detail which you can pattern match on, in a way that you can’t for a short bio.

There’s plenty of talent out there. I think it is part of our job as voters / recruiters to select for talent.

It’s outrageous to complain the country is badly run, if we don’t even do basic due diligence in selecting good people to run it.


Of course, the CV isn’t the be all and end all. It is a minimum. It’s the least possible thing we could look at.

Sometimes I tell people this site is satirical, and they don’t quite get the joke. It’s this:

We, as voters, are interviewing candidates for the £67k job of creating all our laws, and directly running the administration of half our economy.

Yet we don’t routinely do what every employer does for even the lowliest job.

Look at their CV.

If you’ve read this far, I’d love it if you’d help us get more CVs! It’ll only take a couple of minutes. Go to http://cv.democracyclub.org.uk/, put in your postcode, then email and/or tweet all your candidates to ask them to upload your CV.

P.S. My CV has been public and on my website since the 1990s. I have never stood for Parliament.

The advert wars

One of the pitched battles in this century’s cyber war is about advert blocking and injecting.

It’s in full flow.

You can tell – journalist friends complaining that ad blockers have killed Joystiq, a 10 year old gaming magazine; web friends complaining that an ad blocker charges advertisers to not block some ads.

I’ve got some perspective.

Back in 1996, I helped make one of the earliest web advert blockers, WebMask.

Why did we do it?

The early web was fundamentally non-commercial. It was built so as not to be CompuServe or The Microsoft Network. I often bought copies of Adbusters magazine, which unpicked how runaway consumerism harms our culture and our health.

Buy Nothing Day

Our advert blocker was a side project which quickly died. Fascinated with the different technologies that could be used to block ads, for years I maintained the most popular page listing such software.

A little later, I used the traffic from that to give instructions on removing all manner of commercial internet material.

The adverts inside web sites animate, flash and distract. Wouldn’t it be nice to get rid of them? They are not unethical like the other nasties on this page. Indeed, some consider it stealing from the site’s ad revenue to block them. Others either don’t believe in the need for such self sacrifice, object to adverts in principle, or say that since they never click on them anyway no revenue is lost. You decide.

A telling quote.

What’s happening now?

Everyone wants to control our adverts.

Google intercepts our very quest for information to take a sneaky cut – a tax – on everything that we buy online. Facebook uses our addiction as social primates to relationships to sell us corporate brands.

Content creators embed multiple tracking devices, sending our most personal information to complex morasses of robot hucksters.

Display advertising landscape

When the industry finds that people didn’t look at or click the distracting ads any more, they instead place adverts which look like camouflaged tabloid stories.

Or sod it all, just publish fake stories that are actually adverts. After all, what’s it matter, if all the real stories are just retyped PR churnalism?

Real time exchanges auction the pixels on our screen based on our profile, to sell us things we had tried to decide we couldn’t afford to buy. Or just to remind us that we think we’re fat.

My mother rang up the other day asking “what this Bing thing is”. She didn’t like it, not as good as Google. The routine update of some standard software had changed her default search engine. Despite being an experienced user of computers, she had no idea how to change it back.

I can’t even blame this on malware – her web browser’s maker does much the same thing.

Even the hardware manufacturers are at it – Leonovo using their power over our metal to inject new adverts into web pages (risking the security of online banking and shopping in the process).

Such a battle.

What’s the complaint about AdBlock?

As of September there were 144 million active AdBlock users.

It’s no longer a geek thing, like it was when I first blocked ads back in 1996. It’s upsetting some content creators.

In my view, the complaint here is that the users are finally trying to control their adverts. How dare they!

Sorry, but everyone else is trying to control our brains (you saw my list in the section above). Why shouldn’t we try to control them too?

This is a long cold cyberwar. As such it is “zero sum” – nobody is going to do well out of this. To expect me to act morally on behalf of the other combatants during a war has chutzpah, and is a futile expectation.

I’m not going to let advertising companies win the control of our information society by doing whatever they say as if that were morally pure.

Maybe if they had a good business model that helped me out. But they don’t.

Advert companies and advert-funded companies aggregate our private information without our knowing permission. They create insecure data vaults and comms channels, which then governments and criminals easily dip into. They secretly run psychological experiments on our social lives.

If you are going to do whatever you like on your general purpose computer (a server), then I’m going to do whatever I like on my general purpose computer (my laptop). Tough luck.

We can have a truce, but you need to parley first. All sides have to give things up.

Of course it is not that simple. Who controls the power of general purpose computation is one of the key hard challenges of our generation.

We’re not going to solve it in five minutes having a chat on social media. It requires the great casualities, then the concerted dipomacy that led to, say, the Geneva Protocol, which banned chemical weapons.

And even then…  Well, physical war hasn’t ended, has it.

Who is coming to harm in this war?

I spent a long time angsting about investigative journalism – how can it get funded on the Internet?

Digging into the world in detail is vital to society. I’ve tried to help by helping journalists use data – we have many journalist users at ScraperWiki to this day.

Adverts are not the answer – the kind of journalism I want doesn’t drive traffic. Take a look at the celeb soft porn of the (profitable) Mail Online front page. It’s very different from the newstand Daily Mail’s crazed politics front page. Telling as to what kind of journalism Internet advertising funds.

In my view philanthropy is the best business model we’ve got for good, socially worthwhile journalism. Look at the excellent ProPublica for an example.

[ Aside: Perhaps it always was the model – a long time ago in US cities, newspapers got their money from classified ads. They didn’t need to do socially useful exposés of corruption to keep that money, but their owners chose to anyway. They loved their city. ]

The other main victim is the smaller content creators. You can feel their pain in Matthew Hughes’s article about AdBlock on MakeUseOf.

If you use AdBlock, know you are still screwing over hard working people, because you can’t be bothered to be mildly inconvenienced. (Tweet)

I don’t have a great prescription for them. Use technologies that block advert blockers – preferably inviting your readers to unblock your site, or donate instead. Make sure your advertiser is one of the ones that pays AdBlocker (see below). Only show quality adverts. Try (again!) other business models.

My larger scale prescription for creators is – create standards! Have a proper RFC for a protocol for adverts, and get a standard system for distributed payment built into all browsers (BitCoin’s children will get there in the end).

These will reduce the costs, make the relationship more directly between producer and consumer, and get rid of the parasitical, big data optimising tech advert industry.

Forming the protocols would be a good peace conference.


AdBlocker, in its form which exhorts advertisers to pay it to let their adverts through, is providing value.  It is improving the quality of adverts with its acceptable ads criteria. Adverts shouldn’t obscure content, they shouldn’t fill up most of the page, they should be clearly marked.

I don’t think that’s blackmail – on the contrary, I think it is a fascinating step towards the terms which should be in the final truce.

Meanwhile though, some users either doubt that AdBlock itself hasn’t been corrupted by money, or want to carry on in pitched battle. They’ve moved on to forks of it that continue to block all adverts.

Others are more adventurous.

Brett Lempereur, following an argument in the pub in Liverpool, made Sadblock. It is a morally sound ad blocker. When it detects the adverts, it blocks the entire page – so you can no longer be accused of stealing the content.

If you don’t find even that kind enough to article writers, there’s one final ad blocking option. AdNauseum is a funky AdBlock expansion which loads all the adverts invisibly, and quietly fakes that you clicked on them.

Bliss – no adverts, and money goes to the journalists. (Something not quite right here – who’s losing out again?)

Of course, all that is just users taking control.

The ultimate control?

Moving on from it to form a new technology industry, with better business models, and a better heart.

P.S. The collapse of Joystiq which started this recent argument about advert blocking has a final twist. It has been resurrected as part of Engadget.

Which web development tools are commodities?

We’re really bad at thinking about innovation.


To improve my own sense, I’ve been gradually absorbing Simon Wardley’s Value Chain Mapping since first seeing him talk about it a few years ago.

The picture to the right is an example of one of his maps. Each blob is a technology need.

As you go from left to right in the map, the technologies go from custom built, through product to commodities. (You can read more in an introduction by Simon, also see my post about the product/market space).

Anyway… The purpose of this blog post is to assess the state of play of various technologies a developer needs to make a new web application. Do we still have to make it ourselves, or is it a standardised thing?

I’m writing this while trying to build a web application (called MPCV in this post, to collect the CVs of candidates for Parliament). Since I’m doing it essentially for fun, I’ve got a very low tolerance of extra effort, so I’ve been pushing things as far right as I can.

Here goes:

Compute, Power, Web Server: These, and indeed Laptop and the whole of the rest of industrial civilisation, are at the Commodity stage. Or at least way over to the right hand side of Product.

Web hosting: You can just throw PHP scripts into a directory on a shared hosting account, or register a Flask app with a PaaS like Heroku or Google App Engine. This doesn’t feel like a commodity yet – there aren’t standard methods, the offering isn’t as clearly defined as, say, electricity. So kind of mid to right hand side of Product.

Source code: How programmers keep the product of their labour has a long history, but certainly feels like a commodity now. Pretty well everyone uses git for everything, with Github, Atlassian and Microsoft offering very similar hosting services. Left side of Commodity. The linked issue trackers don’t have strong standards yet and are hard to migrate between, so they’re over to the right of Product.

Web clients: With KHTML’s descendents in nearly every browser, IE6 pretty well actually gone, and web browsers in over a billion end user pockets, this is looking pretty good. Add to that the very mature low level libraries like jQuery and Backbone, and it is a place of dreams. Even video is nailed (I use the <video> tag on Redecentralize). They’re a Commodity.

Mobile apps: A warring duopoly, each with an identical feature set but you have to write your app twice. In different languages. I put the development tools for this in the Product side. Hopefully Firefox OS and/or the W3C will somehow force it into Commodity soon. For this reason I’m not worrying about this for MPCV, mobile web only.

Design templates: I’m using Bootstrap because we use it at ScraperWiki, but it has lots of competitors snapping at its heals. This is well over to the right of Product, getting on into Commodity soon.

Email sending: Firmly in Product. SendGrid and Mailgun are popular and work well. But you have to think about it, it is not like water. In some ways it is worse – on old Unix servers back in the low spam days, it was more of a Commodity.

User identity: The likes of Facebook and Google try to grab terrain here, both developers and their users are wary. There are a few products like Stormpath, none that great yet. Mozilla Persona is a tantalisingly close abandoned attempt – it at least met developers’ need to keep responsibility for their own users. In short, this area is still Custom built. Because that’s what everyone does, 2015 and still rolling email confirmations.

Developer identity: Every single one of the other products in this list requires you make a new user account, or these days one for each person working on the project, and enable and backup your 2FA codes. An unexpected frustration for me working on MPCV is that it is a throwaway project – it doesn’t have organisational boundaries yet. Sometimes I literally got blocked not knowing what to name an account. And I had severe limits on how often I wanted to add a credit card subscription for a short project. Heroku’s app store made me even more confused about this, and nobody seems to use “log in with Github”. This is at the Custom built phase.

Encryption: All good sites need to be over HTTPS these days. Getting the certificates is a rip-off Product. The Electronic Frontier Foundation are improving this with free certificates later this year.

Democracy data: For my purposes, mySociety’s Mapit (for postcode to Parliamentary constituency lookup), and DemocracyClub’s YourNextMP (to get the candidates in a constituency) are fantastic. It felt like Commodity territory.

I’m going to update this post as I come across more categories during development.


There are big big problems with identity. Both my identity and the identity of my users are taking up far too much of my time and attention. This feels like a core weakness – I hope Mozilla try again. I would attack here.

There are too many services scattered everywhere. I don’t think either Amazon or Heroku are doing a good job at bringing them together. They, or Google or Microsoft, will eventually.

Mobile apps are an embarrassing disaster. I hold little hope that there is an attack point against this, but who knows.

Promising to make software safer

1. Virtual bugs

I first really knew that all software was fundamentally insecure back in 2001.

I was working for an artificial life games company. We made virtual pets – amazing ones with a simulated brain, biochemistry and genetics.

Creatures Docking Station

I’d just built a new networked version called Creatures Docking Station. It let the cute, furry, egg-laying Norns travel through portals, crossing the Internet directly between player’s computers.

The game engine was built using the language C++. It was fiendishly complicated – neuron simulators for the Norn brains, a scripting language implementing all the in game objects. About 20 people had worked on it, with varying needs, skills and time pressures.

I knew that there were bugs in it. I’d previously stress tested the code – randomly mutating Norns and force breeding them with each other in a diabolical machine, while the game was running in a debugger. It found a new crash bug every hour – I’d tap Gavin on the shoulder and get him to fix each one. We never got them all.

The symptom – the game crashing occasionally due to a mutation – wasn’t itself a world shattering problem. No real lives were on the line. Bad user experience, but so what?

Two reasons:

1) In C++, bugs in this category let an attacker do anything they like. That is, much like a chain saw, with great power comes great responsibility.

2) With the new networked game, it would let an attacker do anything they liked, remotely and automatically from across the network.

In short, a player of our game could have their machine taken over remotely – their documents deleted, spam sent, their Internet banking password sniffed (not that many people used Internet banking back then). Whatever the attacker wanted.

At the time, there was no tool or technology or budget available for me to fix this. I did what every programmer did – closed my eyes. Ignored the problem. Hoped nobody would do bad things with it.

I knew though that nearly all general purpose software, particularly written in C/C++, was likely to be insecure.

2. A simple promise

Wind forward to 2015.

I’ve been worrying for a while about the long, cold cyberwar. A small part of that war is basic security of all computer systems – so it’s hard for criminals or rogue states to, say, remotely turn on your microphone without you knowing.

Linking this to my old experience with C++, and a constant flow of security vulnerabilities which could only happen to C/C++ code, I had the idea that as an industry we should stop using C/C++.

Peter had shown me how good Go is now (we use it a lot at work), making my historical needs for C/C++ now obsolete. Suddenly, it felt possible to completely stop using those languages.

This comment on Hacker News finally provoked me into actually doing something. I knocked up a very simple promise site in an hour on a Sunday afternoon.

I promise never to use C/C++ for a new project

If you’re a programmer, go on, go and sign it!

This seemingly simple promise felt like putting my head over a parapet during a siege. I learnt quite a lot.

3. Things I learnt

Embedded systems – lots of people wouldn’t take the promise because they work on embedded systems where C dominates. Others pointed out various rays of hope – Python for microcontrollers, OCaml for PICs, LLVM for AVR chips, embedded Rust, even Go for Arduinos (OK, not quite!). Are those going to be good enough, even if you have to direct code for cycle goals?

My view is that it is particularly important to sort this out now. Embedded devices are joining the internet more and more – even if you’re writing something which is standalone now, some other programmer will connect it to something in 5 or 10 years. I don’t want my physical devices to be easy to hack into. The pledge I’d really like embedded systems developers to take is to try using and improve on the new more secure toolchains.

C++ is secure now – a few people pointed out that C++14 can now be used with safe pointers and sanitisers. Others have proposed friendly dialects of C where you turn all the safe compiler options on.

In principle I’m up for this, but only if it is forced in an explicit language variant – otherwise someone will shoot themselves in the foot later. I’m not sure it is worth it in most cases, compared to using Go or Rust. Either way, legacy C/C++ code is the really big issue.

Go or Rust flaws – a few people don’t like them, sometimes for aesthetic syntax reasons, sometimes claiming they are hard to use. I don’t think C has a particularly great syntax – I can remember trying to learn it when I was 15, it wasn’t easy. Sure, if you don’t like them, pick something else. It doesn’t mean you have to juggle with chainsaws.

Of course, these new languages still have parts written in C, at least for now. There can always be bugs in their compilers and assemblers. I don’t think this is a big problem, as those parts are a much much smaller surface of attack – albeit a valuable one.

Application binary interfaces – what can we use instead of C as the standard ABI? Pretty well all languages in the “open source” world interoperate with each other via C bindings. If you took my promise, would you still be able to write Python bindings to an existing C library? Pretending we don’t need C is just fantasy.

This is by far the best criticism. Of course, the Java and .NET worlds have spent a decade building entirely new ecosystems which strongly discourage C bindings. So it’s perfectly possible. We will need something specific to use instead. I don’t know what it should be – this needs strong leadership, maybe from the Rust people.