Making our information society safe and fair

The topic of how to make our information society safe and fair regularly comes up in conversations.

I think we need some quite big, radical things. They’ll need new public service Internet organisations to implement.

This is my high level view list.

1. Access to culture

“People have too much knowledge already: it was much easier to manage them twenty years ago; the more education people get the more difficult they are to manage.” (one MP’s response to the Public Libraries Act 1850)

Liverpool Central Library
The printing press wasn’t a fair innovation until centuries later, when Victorians made the public library widespread. Some of the Internet is free at the point of use, due to advertising and due to the free culture movement. Some of it isn’t.

Much of the most detailed, intelligent reporting – such as the Financial Times and the Economist – is not free. Much vital culture – such as the Wire – is not free.

There are numerous business publications and research organisations, which aren’t free.

I think we’ll need a method, similar in purpose to public libraries, which gives poor teenagers access to the same culture as rich ones. Which gives someone with few resources starting a new business access to the same data as a rich corporation.

Where to start: Build on existing physical libraries – make sure they have subscriptions to paid for aspects of research and culture. Build on free culture. Imaginative ideas for new funding methods.

2. Digital literacy

Peasants on a collective farm in Livezeni village, Arge? County being read a newspaper in 1950

Many technology innovations only reach their best when complemented with mass education.

We teach nearly everyone in the UK to read and write. A majority learn to drive. Without that quite complex training, the printing press and the car would be just for an elite.

What could everyone learn which would help us make better use of computers? Usability, which I love, will only get us so far. Until we have strong Artificial Intelligence (it’ll be a while), many people in society will need to understand computers well.

There are still plenty of technology changes to come, and we can’t know exactly what to teach until after they’ve happened. Meanwhile, we can make a start, and iterate as we learn more.

Where to start: Teach children to code. Make sure employees can use spreadsheets in a sophisticated way. Even teach police to type! (Richard Pope‘s idea after seeing a desk sergeant who was very slow at entering his paperwork)

3. Professional programming

Tacoma Narrows bridge

Engineers don’t build bridges that fall down, not any more. Software is constantly broken, our data stolen and our privacy breached.

There’s a whole lot of it which needs rebuilding in new ways which are barely researched yet. The industry needs to be professional to do this.

Ethical policies which help defend privacy. Quality policies which make it secure. Diversity policies which makes it usable for everyone… There are lots of things a professional programming organisation could improve.

I’ve benefited a lot from the accessibility of programming – I learnt as a hobbyist child from my father. We can make programming both accessible, and professional.

Where to start: Look at other engineers. Look at other professions, like doctors and lawyers. Join and improve ethical professional bodies. Consciously try to not harm the freedom which general access to programming gives in the process. Create standards.

4. Building regulations

Building regulations

Increasing skills is important, but won’t be thorough enough. We need to enforce standards by law too.

Some of these will be mundane but vital, such as websites using only a few standard sets of terms and conditions. Others will be life saving, such as making sure your driverless car manufacturer has a high standard of software engineering practice.

Where to start: Regulate to stop coding in unsafe languages. Support organisations like I Am The Cavalry (automotive software safety). Build on industry guidelines until they are mandatory (e.g. MISRA).

5. Ethical cryptography

First woman jury, Los Angeles

It would be foolish to use no encryption, allowing Governments and criminals to spy on everything we do. And completely unethical.

It is just as wrong to hope that all things will be encrypted in a libertarian utopia. There are criminals and enemies who courts should be able to get evidence from.

This blog post by Vinay Gupta describes the three actors which cryptography should model – the users, the Government and criminals. It describes a much more sophisticated threat model than we tend to talk about.

Where to start: Implement ideas like Cheap ID and jury-based crypto. Develop capability to practice a hybrid of law and tech. Use that to develop technical / legal systems similar to constitutions.

6. Power framework

Hal, 2001

There’s a war over who has access to the core ability of computers – to program them in arbitrary ways.

How is what computers do controlled?

To understand the issue, read Cory Doctorow’s article The Coming Civil War over General Purpose Computing. It’s a very complex question.

Where to start: Build on organisations like the Free Software Foundation, Electronic Frontier Foundation and Open Rights Group. Academic research which combines the abstract philosophy of delegated agency with practical user needs.


What do you think needs doing to make our new information society safe and fair? Leave a comment!

Those brief moments when winning seems possible

This mad dash bad mixen fun up down world.

Forces smash hither and fither, 7 billion of us strange qualia,
doing, being human.

Web culture, open source culture… sucked out of academia,
hacking, sharing, making, funning, building. Just late enough
for usability to be just there just cheap enough.

Smashed with.

Political culture… burnt out, tired, so so so good best
world has ever been, so so so broken, media corporate
capitalist socialist thought tangle.

Wonks, meet Geeks.

Geeky wonky wonky geeky geeky wonky geeky wonky geek geek
geek geek wonk wonk wonk wonk.

Only in England. Honourable neutral, civic servile, playful,
statirical. A soup, aglutinating not once but many times –
digitrights faxing, illegal-war voting, political surveying…

Pulled together into an organ, into a community, into a charity.
Inspiring. Beast poking.









Love us hate us join us be us. Pester your politicians, repair
your civic infrastructure, free your information!

there are few of those very brief moments those times when two
things combine in the right place with the right skills and
set the pulse and light up and burn and grow and all the
people and all the amazing people and spread spread from
this island round the world ideas filtering through a million minds
and alter the framing change the expectations uplift uplift
hope hope hope.

Those very brief moments.

When winning seems possible.

“I find politics inaccessible”: User testing voter advice apps

How is she going to vote? My friend (let’s call her F) just doesn’t know.

This isn’t unusual – about 20% of voters are like her (see Q3).

She needs to work it out fast. She doesn’t like polling stations, so has a postal vote. She has to decide this weekend – or she’ll get busy with work, and not remember to post the ballot in time.


mySociety helpfully list 16 voter advice apps, so I decided this was a chance to test them out!

This is a real voter, who doesn’t like politics – or to be precise finds it “inaccessible”. Exactly the kind of person these apps are trying to help. She’s a 30 year old professional, lives in Liverpool.

It’s a pretty harsh bit of user testing, I’m not going to sweeten my report of her reactions to the sites. So here goes… (Skip to the end for a summary if you get bored)

1. Who Should You Vote For? 0/10

We’re on mobile (my Fairphone 1, using Opera browser) – for one thing, we’re on a train and then in a coffee shop. For another, F says she would never do this on her laptop. If she had spare time at home, she’d be more likely to watch Netflix. Voting intention is relegated to spare time while waiting for a bus.

She read the first two questions of Who Should You Vote For? – one about a 50% tax rate for top earners, another about the mansion tax.

Who Should You Vote For questionsThen declared it boring. She doesn’t care about the questions, and there are too many of them.

2. Vote for policies 1/10

F immediately liked being offered a choice of issues that she cared about. She hated the wording on the more detailed questions that followed – didn’t understand them.

Vote for policies

She gave up after the first party. If she could have skipped to the second party she would have, but you can’t click on the numbers to do that.

My view is that the site uses wordings from manifestos, and these just aren’t written in an accessible way.

3. Position dial 0/10

“Pictures? What’s all this stuff? I don’t know what I’m supposed to do.”

Position Dial confusing

She closed the window. The site is just too confusing a mush, without a clear call to action.

4. iSideWith 9/10

F likes the questions. The interface on mobile was “a bit jerky” – for some reason it kept zooming in when choosing answers. There was also no explicit skip button for questions, but she found out that you can just not answer them.

She liked the “more stances” button, although it rarely satisfied her. In one case she actually typed in a new detailed one!


We lost her answers due to the mobile browser and the train. She didn’t mind – reentered them in the coffee shop. She liked it that much.

She got her answer: Lib Dem 98%, Labour 98%, Green 93%, Conservative 65%. She then tried the function to compare results, she wanted to know why the scores varied. What were the important policies? It wasn’t clear.

All told, the high score was because she found the questions comprehensible, of the right volume, and fun to answer.

5. Vote Match 7/10

It feels like a speed dating site!

“Too hard” – dragging the order of topics. F put the first three in order, but then got stuck. She didn’t know what “political power” meant or what “trending issues” meant. Nevertheless, she went to the next stage no problem.

Her first reaction is that she doesn’t like the questions much, but isn’t sure why. The iSideWith ones were just easier to understand. For example – the “find out more” button for the fracking question explains what fracking was, but gives no context as to why it might be good or bad, or what current policy is.

Vote Match

“The Government’s top priority should be too sort out the deficit” – F didn’t like the wording of the question. It is important, but surely top priority should be to keep us safe and not starving! (My view is that this is over use of standard political wording, which is alienating)

“Public spending should be maintained at least at current levels” – F doesn’t understand what “public spending” means – it doesn’t say what spending! Depends what they cut. Isn’t very clear. (Again this is a political class phrasing and argument, meaningless in day to day life)

“UK is better off in the EU” – doesn’t tell F the postivies/negatives. There is no find out more. Wants a list of advantages / disadvantages of being in the EU.

Result: Green 73%, Labour 70%, Lib Dem 63%, UKIP 38%, Conservative 30%.

F’s assessment so far: This is not helping – one quiz said vote Green the other vote Labour! They didn’t say why in either case.

(Later F tried the Telegraph version of this and quickly realised it was the same. She was skeptical, worried Telegraph would be bias.)

6. 1/10

I’ve put in my postcode. It’s telling me what people are. Why? Tells me how to contact them. Hasn’t told me anything about them.

F didn’t see link to “Personal homepage”, she thought it was another contact link.


With prompting, she then tried a website of Green candidate. She found there was too much to read, it was irrelevant blog posts, nothing about policies. Wanted policies.

Neither she nor I noticed the “Choose comparison” dropdown, which lets you compare policies, until I was making this blog post later. It’s a bit clearer on desktop.

7. Tickbox 2/10

Weird that they all want my postcode.

The initial “Have I decided who to vote for” annoyed her – of course not, that’s why she’s using the site!

She’d heard people talking about Trident, knew it was about defense, but didn’t know what it was. She liked the explanation on the site – that it was about
cost vs. security. She likes the explicit “don’t know” button. “GP surgeries 7 days a week” – F was surprised! A new question that none of the other sites had asked.


Result: 1) Tory 2) Lib Dem 3) Labour

She didn’t like the answer, doesn’t believe she’s Tory (doesn’t like them, and other quizes didn’t suggest them). Also it only answered 5 questions. So she goes for 5 more questions.

10 questions in: 1) Lib Dem 2) Labour 3) Tory

No Green! Only 3 parties. Not useful. She likes the explanations of the questions on Tickbox, but just doesn’t believe their results. She kept going to see if they stabilised. Grammar schools – likes them, first time asked! (Clever kids get left behind in current schools, not fair)

15 questions in: 1) Labour 2) Tory 3) Lib Dem

Still doesn’t believe it. Also says that the “5 more questions” button isn’t very forceful, and loads of people would miss it. So get incorrect answers from too few questions.

20 questions in: 1) Tory 2) Labour 3) Lib Dem

“It is just making it up”. Likes questions and explanations, but pointless if it is just giving a rubbish result. She asks a general question – how can I know if these quizzes are bias?

8. Who Shall I Vote For 5/10

Too much text on the first page saying it isn’t bias, F didn’t read it and it annoyed her (!).

She likes the choice of issues again. Although now is worried might miss important things. She realises how hard it is to ask about the right issues in these quizzes. She didn’t want to pick 3 topics, it made her pick 3. Now has to do 10 questions on things she doesn’t care about.

UX pain – couldn’t go back and set scale on first question, when on second question. She doesn’t like the lack of any explanation of the questions.

Yay, first to ask about badgers! Likes that.


There’s then a principles section which asks really general philosophical questions about approach to life and decisions. She finds them really hard to answer, and has to skip loads of them.

Principles section

Male or female – why’s it need to know! (She skimmed and didn’t see the “research purposes” bit). She then refuses to tell it who she would vote for – thinks it may bias it, and that it is none of its business. I try to get clarity on this, but couldn’t – it was very instinctive reaction. Doesn’t want to give postcode, hated that, no reason for site to ask.

Result: 1) Green 2) Labour 3) SNP 4) Plaid 5) Lib Dem

Asked very detailed about 3 things she cared about. She likes breakdown of every question by party.  Overall though, a bit too detailed. Rather be asked more generic questions about fewer topics.

9. WhoGetsMyVoteUK 2/10

Sex/age/education – she hates being asked. The copy says “this app does not collect any personal identifying information”, so she says “why does it need this personal info then”? Skip!


She pressed wrong answer for a question, then the back button goes back to beginning not to the previous question. Rubbish!

Questions seemed OK, but hated losing all her work when trying to go back. Gave up quickly.

She’s getting bored :)

10. Verto 0/10

F doesn’t like being asked where born and where lives. Doesn’t want to answer loads of data collection questions.

This app has left/right swiping like Tinder, which she has never used. She can’t remember which means like and which means dislike! Noticing she’s being slow, the site pops up a helpful dialog.

Screen Shot 2015-04-25 at 19.58.24

That doesn’t clearly explain which direction is which, so she still doesn’t know! Gives up.

Overall view – she likes idea of playing a game, but it took to long to get there with other questions, and then had a confusing swipe interface.

11. Voting Counts Policy Matrix 0/10

F thought could click on Issues like NHS but can’t. Why such big space when not links? She complains she can’t compare breakdown of parties, that it should be in table!

It’s then that we realise the site doesn’t work right on mobile. I show her the desktop version on my laptop, and she likes the look of it – but that isn’t relevant to this test, which is mobile only.

12. Your Democracy 0/10

Got to create an account – not doing that. Can’t be bothered.

Your Democracy

13. Awedience -1/10

This site asked for F’s postcode, when she entered it it said “Unknown analysis”.


She was so annoyed that she’d gone to the effort of giving her postcode, but got nothing back, she gave it a negative score.

14. Election Compass 0/10

There’s a bug on mobile – can only see half the page, and can’t zoom or drag sideways.

15. The Economist 0/10

F at first worries they’re bias. Suspects they’re not, but isn’t sure why she thinks that.

She complains she can’t read it properly – fonts too small on mobile. Then a box pops up, and she can’t read all the content.


She gives up.

16. 0/10

Another site that just tells me about the candidates. Doesn’t help me decide who I’m going to vote for.


F felt it would be more useful a site to someone who is into politics and already knows stuff. She also felt she’d already seen it – same kind of experience as YourCandidates. Overall, boring.

17. Democracy Club CVs 0/10

I decide to test out this site, which I made! F knows it’s mine, but doesn’t hold off on criticism…

She enters her postcode, then says that because not everyone has submitted their CV, it is not representative. It does makes her think they can’t be bothered, so she sees that as a negative.

Not submitted their CV

It doesn’t help her decide, tells her about the person. She wants to know about policies.

I don’t want to vote for XXX because she has good experience, if she also kills badgers!


  • Do user testing. Lots and lots and lots of it.
  • Write the content properly. Don’t use wordings from manifestos, or any wonkish part of the political system.
  • Explain what things mean and current policy. Help the voter relate issues to their own knowledge / opinions.
  • Develop and test on mobile. May as well write it in Swahili if you don’t.
  • Several quizzes were rejected entirely because of small (in code!) UX bugs which made them unusable.
  • Don’t ask personal questions like sex / age / postcode, or even how voted last time. Off putting.
  • When giving results, explain simply why – what policies most distinguish the parties in context of what voter cares about.

The winning app is iSideWith! Second and third place go to Vote Match and Who Shall I Vote For.

Nobody else came close, all scoring 2 or less out of 10.

After all that, F still doesn’t know how to vote. She needs help choosing a criteria to distinguish Labour and Green.

Why I’m collecting every MP candidate’s CV

My side project for the last month is to try and collect the Curriculum Vitae of everyone standing for Parliament.

It’s called Democracy Club CVs.

I’ve been working every spare hour – mainly around midnight and on Sundays. Partly it is technically interesting, partly the other Democracy Club volunteers are fun to hang out with…

It’s also just enjoyable, with leaflets from ex-whips, professional musicians and raving loony leaders (that last one is standing against Boris).

But that doesn’t really explain … why am I doing it?

Background matters

The idea for the site came from this election leaflet Julian found in the 2010 election (Guardian article). It has a kind of CV written by Jacob Rees-Mogg’s opponent. Take a look.

Rees-Mogg CV Election Leaflet

I don’t have a fixed idea of what background makes a good MP. I do, however, like the idea (thanks James) that their background should be representative of their constituents.

Hopefully, when we have enough CVs, academics linked to Democracy Club can start doing this kind of analysis.

Layout and form

Aine, one of the directors at ScraperWiki, really likes CVs. It’s not that they’re magical things that tell you everything you need to know about a job applicant. Why is it?

Firstly, the form of the CV tells you quite a bit.

Is it clearly laid out, easy to read, spelt correctly? Is it evidently a CV they’ve used to apply for other jobs recently, or have they never had to apply for a job and it is just their political bio recanned?

It’s not just me that thinks this matters – we ran some user testing with unconnected members of the public, and it was the main thing they judged the CVs on.

To try and encourage good formats, we added this to the upload page.

This is a CV

In later desperation, I also gave a link to the National Careers Service advice on writing a CV.

Skills and talents

Secondly, the content gives some indication of skills.

Would our Prime Ministers like a better talent pool to pick ministers from? Would it increase quality in the way the country is run? I don’t know!

But it wouldn’t hurt to try!

You can pick a maths teacher,  a mental capacity lawyer, a manufacturing engineer, an expert in changing complex organisations, a carbon capture lead… And that’s just from the few CVs we have in already.

With a CV the details tell you lots – the exact list of places and names of companies, whether there are gaps. The progression in job titles. It’s all that detail which you can pattern match on, in a way that you can’t for a short bio.

There’s plenty of talent out there. I think it is part of our job as voters / recruiters to select for talent.

It’s outrageous to complain the country is badly run, if we don’t even do basic due diligence in selecting good people to run it.


Of course, the CV isn’t the be all and end all. It is a minimum. It’s the least possible thing we could look at.

Sometimes I tell people this site is satirical, and they don’t quite get the joke. It’s this:

We, as voters, are interviewing candidates for the £67k job of creating all our laws, and directly running the administration of half our economy.

Yet we don’t routinely do what every employer does for even the lowliest job.

Look at their CV.

If you’ve read this far, I’d love it if you’d help us get more CVs! It’ll only take a couple of minutes. Go to, put in your postcode, then email and/or tweet all your candidates to ask them to upload your CV.

P.S. My CV has been public and on my website since the 1990s. I have never stood for Parliament.

The advert wars

One of the pitched battles in this century’s cyber war is about advert blocking and injecting.

It’s in full flow.

You can tell – journalist friends complaining that ad blockers have killed Joystiq, a 10 year old gaming magazine; web friends complaining that an ad blocker charges advertisers to not block some ads.

I’ve got some perspective.

Back in 1996, I helped make one of the earliest web advert blockers, WebMask.

Why did we do it?

The early web was fundamentally non-commercial. It was built so as not to be CompuServe or The Microsoft Network. I often bought copies of Adbusters magazine, which unpicked how runaway consumerism harms our culture and our health.

Buy Nothing Day

Our advert blocker was a side project which quickly died. Fascinated with the different technologies that could be used to block ads, for years I maintained the most popular page listing such software.

A little later, I used the traffic from that to give instructions on removing all manner of commercial internet material.

The adverts inside web sites animate, flash and distract. Wouldn’t it be nice to get rid of them? They are not unethical like the other nasties on this page. Indeed, some consider it stealing from the site’s ad revenue to block them. Others either don’t believe in the need for such self sacrifice, object to adverts in principle, or say that since they never click on them anyway no revenue is lost. You decide.

A telling quote.

What’s happening now?

Everyone wants to control our adverts.

Google intercepts our very quest for information to take a sneaky cut – a tax – on everything that we buy online. Facebook uses our addiction as social primates to relationships to sell us corporate brands.

Content creators embed multiple tracking devices, sending our most personal information to complex morasses of robot hucksters.

Display advertising landscape

When the industry finds that people didn’t look at or click the distracting ads any more, they instead place adverts which look like camouflaged tabloid stories.

Or sod it all, just publish fake stories that are actually adverts. After all, what’s it matter, if all the real stories are just retyped PR churnalism?

Real time exchanges auction the pixels on our screen based on our profile, to sell us things we had tried to decide we couldn’t afford to buy. Or just to remind us that we think we’re fat.

My mother rang up the other day asking “what this Bing thing is”. She didn’t like it, not as good as Google. The routine update of some standard software had changed her default search engine. Despite being an experienced user of computers, she had no idea how to change it back.

I can’t even blame this on malware – her web browser’s maker does much the same thing.

Even the hardware manufacturers are at it – Leonovo using their power over our metal to inject new adverts into web pages (risking the security of online banking and shopping in the process).

Such a battle.

What’s the complaint about AdBlock?

As of September there were 144 million active AdBlock users.

It’s no longer a geek thing, like it was when I first blocked ads back in 1996. It’s upsetting some content creators.

In my view, the complaint here is that the users are finally trying to control their adverts. How dare they!

Sorry, but everyone else is trying to control our brains (you saw my list in the section above). Why shouldn’t we try to control them too?

This is a long cold cyberwar. As such it is “zero sum” – nobody is going to do well out of this. To expect me to act morally on behalf of the other combatants during a war has chutzpah, and is a futile expectation.

I’m not going to let advertising companies win the control of our information society by doing whatever they say as if that were morally pure.

Maybe if they had a good business model that helped me out. But they don’t.

Advert companies and advert-funded companies aggregate our private information without our knowing permission. They create insecure data vaults and comms channels, which then governments and criminals easily dip into. They secretly run psychological experiments on our social lives.

If you are going to do whatever you like on your general purpose computer (a server), then I’m going to do whatever I like on my general purpose computer (my laptop). Tough luck.

We can have a truce, but you need to parley first. All sides have to give things up.

Of course it is not that simple. Who controls the power of general purpose computation is one of the key hard challenges of our generation.

We’re not going to solve it in five minutes having a chat on social media. It requires the great casualities, then the concerted dipomacy that led to, say, the Geneva Protocol, which banned chemical weapons.

And even then…  Well, physical war hasn’t ended, has it.

Who is coming to harm in this war?

I spent a long time angsting about investigative journalism – how can it get funded on the Internet?

Digging into the world in detail is vital to society. I’ve tried to help by helping journalists use data – we have many journalist users at ScraperWiki to this day.

Adverts are not the answer – the kind of journalism I want doesn’t drive traffic. Take a look at the celeb soft porn of the (profitable) Mail Online front page. It’s very different from the newstand Daily Mail’s crazed politics front page. Telling as to what kind of journalism Internet advertising funds.

In my view philanthropy is the best business model we’ve got for good, socially worthwhile journalism. Look at the excellent ProPublica for an example.

[ Aside: Perhaps it always was the model – a long time ago in US cities, newspapers got their money from classified ads. They didn’t need to do socially useful exposés of corruption to keep that money, but their owners chose to anyway. They loved their city. ]

The other main victim is the smaller content creators. You can feel their pain in Matthew Hughes’s article about AdBlock on MakeUseOf.

If you use AdBlock, know you are still screwing over hard working people, because you can’t be bothered to be mildly inconvenienced. (Tweet)

I don’t have a great prescription for them. Use technologies that block advert blockers – preferably inviting your readers to unblock your site, or donate instead. Make sure your advertiser is one of the ones that pays AdBlocker (see below). Only show quality adverts. Try (again!) other business models.

My larger scale prescription for creators is – create standards! Have a proper RFC for a protocol for adverts, and get a standard system for distributed payment built into all browsers (BitCoin’s children will get there in the end).

These will reduce the costs, make the relationship more directly between producer and consumer, and get rid of the parasitical, big data optimising tech advert industry.

Forming the protocols would be a good peace conference.


AdBlocker, in its form which exhorts advertisers to pay it to let their adverts through, is providing value.  It is improving the quality of adverts with its acceptable ads criteria. Adverts shouldn’t obscure content, they shouldn’t fill up most of the page, they should be clearly marked.

I don’t think that’s blackmail – on the contrary, I think it is a fascinating step towards the terms which should be in the final truce.

Meanwhile though, some users either doubt that AdBlock itself hasn’t been corrupted by money, or want to carry on in pitched battle. They’ve moved on to forks of it that continue to block all adverts.

Others are more adventurous.

Brett Lempereur, following an argument in the pub in Liverpool, made Sadblock. It is a morally sound ad blocker. When it detects the adverts, it blocks the entire page – so you can no longer be accused of stealing the content.

If you don’t find even that kind enough to article writers, there’s one final ad blocking option. AdNauseum is a funky AdBlock expansion which loads all the adverts invisibly, and quietly fakes that you clicked on them.

Bliss – no adverts, and money goes to the journalists. (Something not quite right here – who’s losing out again?)

Of course, all that is just users taking control.

The ultimate control?

Moving on from it to form a new technology industry, with better business models, and a better heart.

P.S. The collapse of Joystiq which started this recent argument about advert blocking has a final twist. It has been resurrected as part of Engadget.

Which web development tools are commodities?

We’re really bad at thinking about innovation.


To improve my own sense, I’ve been gradually absorbing Simon Wardley’s Value Chain Mapping since first seeing him talk about it a few years ago.

The picture to the right is an example of one of his maps. Each blob is a technology need.

As you go from left to right in the map, the technologies go from custom built, through product to commodities. (You can read more in an introduction by Simon, also see my post about the product/market space).

Anyway… The purpose of this blog post is to assess the state of play of various technologies a developer needs to make a new web application. Do we still have to make it ourselves, or is it a standardised thing?

I’m writing this while trying to build a web application (called MPCV in this post, to collect the CVs of candidates for Parliament). Since I’m doing it essentially for fun, I’ve got a very low tolerance of extra effort, so I’ve been pushing things as far right as I can.

Here goes:

Compute, Power, Web Server: These, and indeed Laptop and the whole of the rest of industrial civilisation, are at the Commodity stage. Or at least way over to the right hand side of Product.

Web hosting: You can just throw PHP scripts into a directory on a shared hosting account, or register a Flask app with a PaaS like Heroku or Google App Engine. This doesn’t feel like a commodity yet – there aren’t standard methods, the offering isn’t as clearly defined as, say, electricity. So kind of mid to right hand side of Product.

Source code: How programmers keep the product of their labour has a long history, but certainly feels like a commodity now. Pretty well everyone uses git for everything, with Github, Atlassian and Microsoft offering very similar hosting services. Left side of Commodity. The linked issue trackers don’t have strong standards yet and are hard to migrate between, so they’re over to the right of Product.

Web clients: With KHTML’s descendents in nearly every browser, IE6 pretty well actually gone, and web browsers in over a billion end user pockets, this is looking pretty good. Add to that the very mature low level libraries like jQuery and Backbone, and it is a place of dreams. Even video is nailed (I use the <video> tag on Redecentralize). They’re a Commodity.

Mobile apps: A warring duopoly, each with an identical feature set but you have to write your app twice. In different languages. I put the development tools for this in the Product side. Hopefully Firefox OS and/or the W3C will somehow force it into Commodity soon. For this reason I’m not worrying about this for MPCV, mobile web only.

Design templates: I’m using Bootstrap because we use it at ScraperWiki, but it has lots of competitors snapping at its heals. This is well over to the right of Product, getting on into Commodity soon.

Email sending: Firmly in Product. SendGrid and Mailgun are popular and work well. But you have to think about it, it is not like water. In some ways it is worse – on old Unix servers back in the low spam days, it was more of a Commodity.

User identity: The likes of Facebook and Google try to grab terrain here, both developers and their users are wary. There are a few products like Stormpath, none that great yet. Mozilla Persona is a tantalisingly close abandoned attempt – it at least met developers’ need to keep responsibility for their own users. In short, this area is still Custom built. Because that’s what everyone does, 2015 and still rolling email confirmations.

Developer identity: Every single one of the other products in this list requires you make a new user account, or these days one for each person working on the project, and enable and backup your 2FA codes. An unexpected frustration for me working on MPCV is that it is a throwaway project – it doesn’t have organisational boundaries yet. Sometimes I literally got blocked not knowing what to name an account. And I had severe limits on how often I wanted to add a credit card subscription for a short project. Heroku’s app store made me even more confused about this, and nobody seems to use “log in with Github”. This is at the Custom built phase.

Encryption: All good sites need to be over HTTPS these days. Getting the certificates is a rip-off Product. The Electronic Frontier Foundation are improving this with free certificates later this year.

Democracy data: For my purposes, mySociety’s Mapit (for postcode to Parliamentary constituency lookup), and DemocracyClub’s YourNextMP (to get the candidates in a constituency) are fantastic. It felt like Commodity territory.

I’m going to update this post as I come across more categories during development.


There are big big problems with identity. Both my identity and the identity of my users are taking up far too much of my time and attention. This feels like a core weakness – I hope Mozilla try again. I would attack here.

There are too many services scattered everywhere. I don’t think either Amazon or Heroku are doing a good job at bringing them together. They, or Google or Microsoft, will eventually.

Mobile apps are an embarrassing disaster. I hold little hope that there is an attack point against this, but who knows.

Promising to make software safer

1. Virtual bugs

I first really knew that all software was fundamentally insecure back in 2001.

I was working for an artificial life games company. We made virtual pets – amazing ones with a simulated brain, biochemistry and genetics.

Creatures Docking Station

I’d just built a new networked version called Creatures Docking Station. It let the cute, furry, egg-laying Norns travel through portals, crossing the Internet directly between player’s computers.

The game engine was built using the language C++. It was fiendishly complicated – neuron simulators for the Norn brains, a scripting language implementing all the in game objects. About 20 people had worked on it, with varying needs, skills and time pressures.

I knew that there were bugs in it. I’d previously stress tested the code – randomly mutating Norns and force breeding them with each other in a diabolical machine, while the game was running in a debugger. It found a new crash bug every hour – I’d tap Gavin on the shoulder and get him to fix each one. We never got them all.

The symptom – the game crashing occasionally due to a mutation – wasn’t itself a world shattering problem. No real lives were on the line. Bad user experience, but so what?

Two reasons:

1) In C++, bugs in this category let an attacker do anything they like. That is, much like a chain saw, with great power comes great responsibility.

2) With the new networked game, it would let an attacker do anything they liked, remotely and automatically from across the network.

In short, a player of our game could have their machine taken over remotely – their documents deleted, spam sent, their Internet banking password sniffed (not that many people used Internet banking back then). Whatever the attacker wanted.

At the time, there was no tool or technology or budget available for me to fix this. I did what every programmer did – closed my eyes. Ignored the problem. Hoped nobody would do bad things with it.

I knew though that nearly all general purpose software, particularly written in C/C++, was likely to be insecure.

2. A simple promise

Wind forward to 2015.

I’ve been worrying for a while about the long, cold cyberwar. A small part of that war is basic security of all computer systems – so it’s hard for criminals or rogue states to, say, remotely turn on your microphone without you knowing.

Linking this to my old experience with C++, and a constant flow of security vulnerabilities which could only happen to C/C++ code, I had the idea that as an industry we should stop using C/C++.

Peter had shown me how good Go is now (we use it a lot at work), making my historical needs for C/C++ now obsolete. Suddenly, it felt possible to completely stop using those languages.

This comment on Hacker News finally provoked me into actually doing something. I knocked up a very simple promise site in an hour on a Sunday afternoon.

I promise never to use C/C++ for a new project

If you’re a programmer, go on, go and sign it!

This seemingly simple promise felt like putting my head over a parapet during a siege. I learnt quite a lot.

3. Things I learnt

Embedded systems – lots of people wouldn’t take the promise because they work on embedded systems where C dominates. Others pointed out various rays of hope – Python for microcontrollers, OCaml for PICs, LLVM for AVR chips, embedded Rust, even Go for Arduinos (OK, not quite!). Are those going to be good enough, even if you have to direct code for cycle goals?

My view is that it is particularly important to sort this out now. Embedded devices are joining the internet more and more – even if you’re writing something which is standalone now, some other programmer will connect it to something in 5 or 10 years. I don’t want my physical devices to be easy to hack into. The pledge I’d really like embedded systems developers to take is to try using and improve on the new more secure toolchains.

C++ is secure now – a few people pointed out that C++14 can now be used with safe pointers and sanitisers. Others have proposed friendly dialects of C where you turn all the safe compiler options on.

In principle I’m up for this, but only if it is forced in an explicit language variant – otherwise someone will shoot themselves in the foot later. I’m not sure it is worth it in most cases, compared to using Go or Rust. Either way, legacy C/C++ code is the really big issue.

Go or Rust flaws – a few people don’t like them, sometimes for aesthetic syntax reasons, sometimes claiming they are hard to use. I don’t think C has a particularly great syntax – I can remember trying to learn it when I was 15, it wasn’t easy. Sure, if you don’t like them, pick something else. It doesn’t mean you have to juggle with chainsaws.

Of course, these new languages still have parts written in C, at least for now. There can always be bugs in their compilers and assemblers. I don’t think this is a big problem, as those parts are a much much smaller surface of attack – albeit a valuable one.

Application binary interfaces – what can we use instead of C as the standard ABI? Pretty well all languages in the “open source” world interoperate with each other via C bindings. If you took my promise, would you still be able to write Python bindings to an existing C library? Pretending we don’t need C is just fantasy.

This is by far the best criticism. Of course, the Java and .NET worlds have spent a decade building entirely new ecosystems which strongly discourage C bindings. So it’s perfectly possible. We will need something specific to use instead. I don’t know what it should be – this needs strong leadership, maybe from the Rust people.

Long, cold cyberwar


Let’s keep this post simple.

We’re near the start of a long cold, cyber war. Many things make this clear – from Stuxnet to Snowden, from the Sony hacks to Chinese DNS poisoning.

This is a hard time to be in information technology.

Just in raw, technical security terms this is tough – rebuilding every layer of computing infrastructure so that it is safe.

And that’s the easy problem.

The hard problems are emotionally and politically challenging: We have to prevent automated privacy invasion from creating new powerful fascist states. We have to keep the Internet competitive and innovative – a positive creative force.

To give a hint at how hard it is, here are three harsh yet promising articles on key subjects.

I don’t know how long this war will take. I’d prepare for, say, a century.

It took many times longer than that after the invention of the printing press for everyday ideas like copyright, the novel, universal literacy and the public library to settle down.

If it feels tough, that’s because it should be.

If, like me, you’re a programmer, the days of rainbows and unicorns are gone. It’s now about moral responsibility, professional integrity and the strategic creation of new concepts.

Let’s get to it.

I blinked and missed 6 exciting things in the last 20 years of space

I blinked.

A long, slow, twenty year blink.

And meanwhile, space exploration went… Phoooom!

From a distance it looks bad. We haven’t sent humans to the moon for over forty years. There’s no grand, visible, memorable showpiece – apart from space shuttles exploding and being decomissioned.

And yet, when I recently got interested again, I found a flurry of things had happened. Many I had seen in passing, but not really looked at. All together, they add up to something amazing.

1. Space telescopes

It’s simple. Without the atmosphere in the way, you get better pictures. Beautiful pictures.

Much-loved space telescope Hubble reached orbit in 1990. You’ll have seen its iconic photo of the Eagle nebula pillars.

But have you seen this one of young stars sparkling into life 20,000 light-years away?

There are endless more. I get lost reading the Atlas of Peculiar Galaxies and looking at Hubble’s top 100 images. Trying to imagine what they all really are.

2. Space station

You can’t even say this new space world excludes humans.

Since 1998, a station has orbited above us. Continuously inhabited.

Recently, the Internet gives this a new intimacy. You can follow Reid Wiseman, tweeting pretty constantly from space. 3 million people have watched this tour of the station by a departing commander.

Most immediate of all, the HDEV video project gives continuous pictures of the earth from the space station. Through your tablet you can stare, imagining you’re up there, looking down on our planet during a break. Every hour or two, snaking around the world.

3. Gamma-ray bursts

Every now and again we detect a fantastic, crazily strong pulse of energy, as much as the sun emits in its entire 10 billion year lifetime. Nobody knows what they are – perhaps neutron stars colliding with black holes?

To investigate this, astronomers made a series of increasingly powerful satellites, culminating in Swift, which was launched in 2004 and is still running now.

About once a day, Swift detects a Gamma-ray burst and sends its location out via the Gamma-ray Burst Coordinates Network. Immediately, telescopes all over earth and space swivel towards the burst, capturing its afterglow to try and learn more about it.

We don’t have very good photos of them yet. Although there are some gorgeous artists impressions.

The nearest compelling real picture is this one of a Wolf-Rayet star, which are a possible cause of the bursts.

4. Roving on Mars

You can hardly have missed that two chirpy robots have been wondering all over Mars since 2004.

This year, the remaining one finally outpaced the record for longest distance travelled on another world, previously held by a 1970s Soviet moon rover.

They’ve made major scientific discoveries about the atmosphere and geology of Mars.

For example, we now know lots about the water on Mars (there are even polar ice caps!).

More immediately, though, they’ve taken detailed panoramic travel photos of another world. For example, Opportunity snapped this one of the Victoria Crater. Hover over it and scroll right to see more.

5. Landing on a comet

It’s impossible to describe the Rosetta probe’s crazy journey, richocheting off four planets to gain speed. You’ll just have to watch this video.

Ten years in, Rosetta is now in a kind of jolty orbit round a comet. Comets are important as they may contain rock similar to in the early solar system, and they may have been the source of water on earth.

In this photo taken by Rosetta, you can see the dumbbell-shaped comet, and the faint jet of the comet’s tail which is just firing up as the comet heads nearer to the sun.

This month, Rosetta is going to release a baby probe which will land on the comet.

6. Baby universe

Maybe you’re tired of looking at the 12 billion year old galaxies in Hubble’s picture of the deep field, and want something a bit older.

Radiation from shortly after the universe begin, travelling for 13.7 billion years, finally reaches us each day. It’s from a time before stars, when the universe was a cloud of hot gas.

We’ve known about this “cosmic microwave background” radiation since the 1960s. It’s key evidence for the big bang.

Not content with low detail pictures, cosmologists have made a series of spacecraft. WMAP was launched in 2001. It lived way out, cuddled in a gravity well just between the earth and the sun.

After 9 years of observations, it produced this image. The colours represent temperature fluctuations which then grew to become galaxies.

Seeds in a photo of our baby universe.

An email to Nicholas

Dear Nicholas,

Thank you for your previous two letters. I’m sorry I was so slow getting back to you after the first one, that you had to write another.

I didn’t know Canon meant essentially the same things as Round. I’m sure I must have been told, but I never got what it meant or cared. I only really appreciated rounds at all in actually singing them with people at Kentwell (the Tudor recreation thing I do).

Amusingly, I look up the most complex round I know (it isn’t instant to learn, as the note of “well” sounds off, so you have to teach it to drunk people carefully), which is the original “cat is in the well”. It’s called Ding Dong Bell in Ravenscroft’s Pammelia. Where, so I just now amusingly founded, headed “Canons in the unison”.

While trying to find out what exactly round means just now, I came across what glees were originally, and glee clubs. I really wish they still existed!

The ground bass is clearly an important reason that I like Pachelbel‘s Canon. And it doesn’t vary in volume. Oh, and there are hardly any instruments – I think adding any more is basically a waste of time for me, as I’ll fail to pick them out.

Moving on to your second letter… It started to irk me right at the beginning – only at the end did you give me an easy way to articulate why. I wasn’t moved by Williams musical version of Christina Rossetti’s poem. Worse, I wasn’t even moved by the poem itself!

Doing as you say and describing my emotional reactions the first time I heard it…

The voice was irritating, overly oscillating such that I couldn’t pick out the words. It actually managed to make the poem harder to understand. There were some uplifting bits musical in the middle, but the tedium of the vocal parts overruled that.

As for the poem, my! It glazes my eyes over, making me simply not want to read it. It is full of metaphors that have no meaning to me. To such an extent that I’d have to force myself to read it as whipped homework to get anywhere further with it at all.

I am going to take your advice, to not try to “understand” music, and not do so :)

I agree with you that over analysis and understanding can defeat the joy of music. What it can do though, is breakdown practical barriers. I’d like a music recommendation service which could say “don’t bother Francis with Wagner, basically nobody with your low volume range of hearing ever ends up liking it particularly”.

For people who are good at music, and/or who have fallen deeply into one genre pool they can’t see out, these barriers are as fleas to a giant. To those in old people’s homes, or whose voices have just broken, or even who are deaf, and have had music torn for them often unknowingly… They are so important!

To slightly shift subject, I just got back from Bearded Theory. Three relevant musical observations from it:

  • To our surprise, we loved the ambient tent, at the right moments. Not being about love or sex was such a relief, the wilful suspension of the primate social, abandoned for rhythm, the raw dance. e.g. The Orb.
  • Revived acts, from The Stranglers to UB40, were just irritating. They had the odd song you knew, but they weren’t the same as when they were young, and if you didn’t like them already, you weren’t going to by seeing them live. This alone makes it worth supporting new acts, despite the cornucopia of amazing historic music we have at a click now. (Ironic, that contradicted by me liking for the first time ancient The Orb above!).
  • It’s fun playing the Ukelele and/or singing (or Kazooing along). Beardy Keef did a jam, managing to get half the famous musicians on site to turn up too. My strumming sucked, and I couldn’t instantly remember chord patterns after the first verse (they were unlabelled on the second)… But still, that Uke, it brings down barriers. Easier than a recorder or a piano to learn to that important stage of “have fun with” by far.
  • The Monster Ceilidh Band are great.

So yeah, I don’t need sophisticated analysis of music. (Although the part of me that wants to understand consciousness, and suspects music is a vital hack on our brains that will reveal a lot about them, is curious.)

Instead, I want analysis so people can have fun, without being put off by usability barriers that there are gorgeous ways round.

But there is a danger that we become distracted by such intellectual diversions in a similar way that one might become fixated by the form of a Sonnet while missing its meaning

It works both ways. To return to Dr LJ’s Tweet… Is everyone, even just in England, actually hearing Beethoven’s 9th? What’s the most efficient way to make that possible, in the cases where they would like it but just don’t have a way of getting to it?

Coincidentally I was at Bearded Theory with a music therapist (there are very relevant links to papers and things on the News and Downloads page!). Singing war solidarity songs to people with dementia… Makes sense to me.

And alas you need research, like in the paper Dr LJ linked to, to stand a chance at knowing how much to spend on nursing and how much on music.

Best wishes,