“What we need is democracy, not government by secret algorithm.” Cori Crider, director of the technology justice campaign Foxglove
The news that the Home Office is finally going to scrap their visa application decision-making algorithm – because it appears to be taking racist decisions – is some confirmation of the case I made in my new-ish book Tickbox.
The idea that if you simply feed all your institution’s decisions into your software – when those decisions were themselves racially biased, consciously or unconsciously – then nobody should be surprised at what happens. We have known that for some years now. Now the Home Office knows it too.
The main worry about the whole case is that Sopra Steria, the French contractor to the Home Office immigration service – and my nominee for Tickbox of the Year, a so far non-existent award that I would nonetheless like to encourage for their dysfunctional £91 million contract to provide visa services – has just be awarded the contract to process passport applications.
I don’t blame them for the algorithms, but the French IT solutions company took on the contract in 2018 when I was still researching my book. A closer examination of the way they work goes some way to explain how the tickbox system – the mixture of online tickboxes, KPIs, targets and centralised controls that their managers rely on to monitor performance – works so badly for those unlucky enough to require its services.
I had hoped to talk to people from the Home Office immigration service when I was writing the book, but nobody would talk to me. Since then, I have talked to a number of those working for Sopra Steria.
The company was taken on in the aftermath of the Windrush Scandal, which left some people drained, bankrupt or deported, despite spending most of their lives in the UK perfectly legally. They provide services to applicants at 60 centres around the country, helping them get their papers together, copying them and sending them to the Home Office.
The centre they run next to Lunar House, the notorious immigration service headquarters in Croydon, sees up to 900 people a day. You can see them queuing up outside, come rain come heat.
What they need to know is that those employed by Sopra Steria are, despite appearances, emphatically not on their side.
They are forbidden by Home Office rules from telling applicants what is wrong with their application. Their employers also discourage staff from telling them that papers are missing, in case they cancel their appointments and mess up the schedule.
Like most outsourcers, they don’t employ enough staff to deal with the demand, which means that existing staff – or agency workers – get more inflexible pressure on them to work harder, which exacerbates the staffing problems. When they first opened, staff were contracted until 8pm, but the final appointments of the day – which could sometimes take hours – were timed for 7.45pm.
The question of what documents applicants need for different visas – to prove, for example, that their business makes money – but the Home Office refers people to Sopra Steria for advice on this, and vice versa (a £2.50-a-minute helpline).
It is Sopra Steria’s task to verify their scanned documents. This can be time-consuming, so they have introduced a button that says ‘Verify all documents’, which they are encouraged to press once they have checked the ‘mandatory documents’ – the passport and so on. Which means that the rest of the documents are not checked and the application may be rejected just because their scans were not adequate.
And they won’t be told the reason either.
All this goes some way to explaining how Windrush generation applicants could have been treated with the kind of boneheaded ‘computer says no’ which so staggered the Commons committee which investigated last year.
You can see why the Home Office dreams of tickbox solutions. It means nobody has to actually take a human decision. Nobody has to take responsibility either. It displays all the hallmarks of the horror of action which I outlined in my Radix pamphlet Whatever Happened to Doing?
But that is the problem, of course. If we hand over our decisions to algorithms, we will simply become their slaves – and, if they are based on previous decisions by an institutionally racist organisation – some people will continue to face discrimination.
Of course, Sopra Steria is not the only example of tickbox in action. Two more, which I nominated for my non-existent award in the Evening Standard, might include the parcels company myHermes which – for the purposes of this article – will have to stand for all those organisations which so believe in the infallibility of their own systems that there is no way to bypass them or talk to a human being when things go wrong – short of emailing the managing director.
Which is why the myHermes MD has a whole department to answer the desperate.
Or British Gas for their homecare insurance system which, luckily I can simply stop paying, in the hope that they get the message which they seem unable to hear by helpline.
Most of us know precisely what is wrong with Tickbox – that most of these measures or targets either miss the point or get finessed by managers.
Those who can’t see it tend to be the elite forces who run the world – and who believe what they are told by the frontline. And who dream of automated systems that can manage organisations without what they fear are messy human interventions or decisions.
The result is that officials tick boxes to allow them to move on but where nothing has changed (as they seem to have done at Grenfell Tower), or we get tickbox targets which focus on the wrong things because what we really want – love, care, education – isn’t measurable.
Or we find ourselves in a ridiculous complaints loop where is is impossible to talk to a real person who would understand our problem in seconds.
Or we get irritated by being asked on a five-point Likert scale how we would rate our latest minor interaction with the bank – where the rep ‘suggests’ that you might make it a 5.
To tackle this, we first need to name the problem – which is why I wrote the book. Tickbox is the machine those who run the world have created to manage things in their absence: we need to call it out.
Especially for the sake of those trapped inside the ones run by the Home Office, myHermes or British Gas.