When Smart Meters go wrong

Most people don’t think much about firmware – the embedded software which runs the microcontrollers in all of the devices we have around us.  We’re aware of the frustration when they don’t do what they’re meant to, at which point we realise that “smart” may not have been the best adjective to use to promote the product, but even when they do go wrong, turning them off and on again, or taking the battery out generally clears the problem.  They almost always go wrong because the design process didn’t include enough testing, or not enough time was given over to thinking about the “edge cases” – those unexpected combinations of events which result in things not working the way they should.  Most of the time it’s just a short-term annoyance; if it’s worse than that we’ll probably send it back, or throw it out and buy a new one.

However, we do expect safety critical devices like cars and planes and national infrastructure to be a lot better designed than this.  Your boiler turning off because it thinks there’s a flow problem when there isn’t is annoying (time for a firmware upgrade please, Vailant), but it’s not life threatening.  In contrast, a self-driving car that runs over a cyclist is not something the public is generally happy about.  Nor is a plane falling out of the sky.  But where would you put a smart meter in the scale of things that might affect your life?  Last week we found out, and it’s not a happy answer.

At this point it’s worth considering how important a smart meter is?  I’ve tried to have that conversation with various parts of the industry and it’s always a question which has raised eyebrows, because they don’t seem to understand why I’m asking it.  The current line from Smart Energy GB is that “Smart meters can’t solve climate change on their own, but with the smarter, more energy efficient grid they help to create, they are a start”.  At which point they regale us with photos of polar bears and cute animals.  Look closely at the photos and you’ll see that smart meters have also managed to get rid of plastic straws, as there aren’t any of those in the adverts.  Clever chaps down there in Bloomsbury, ensuring the area maintains its reputation for fiction.  Away from the glare of the publicity lighthouse, the Energy Minister of the day tends to trot out the same line that they’re leading to a more efficient energy grid, whilst our energy suppliers enthuse about the fact we’ll no longer get estimated bills.  Or meter readers.

That dichotomy between smart grid and accurate bills is key to understanding the GB smart metering programme and why it’s gone wrong.  The key reason for installing smart meters should be to provide data to make the grid more efficient.  To make the grid efficient, you need to be able to react to demand, which means real-time information and the knowledge of how to use it.  However, in Britain, we have let the meter design be driven by the energy suppliers.  They have no real interest in real-time data; as for billing they only need it on a monthly basis.  Instead, they compromised and designed meters which upload data once a day.  The whole of the rest of the smart metering infrastructure, from the DCC through to the cellular contracts for uploading the meter data, has been designed and costed on the same basis, which means that the £20 billion or so we’re spending on the program will not help us get a smarter, more efficient energy grid.

To be fair, we’re not alone.  Spain has garnered acclaim for rolling out smart meters to every household by the end of 2017.  They made some good decisions, such as not making the specification overly complex and letting the network install them instead of the utilities.  Both decisions helped them to keep their deployment on track at a fraction of the cost of the GB programme.  It will also help them keep their bills down, as the cost of smart meters ends up being paid for by consumers.  However, a recent set of interviews with senior managers involved with the project shows a lot of them wondering whether there will be an overall cost benefit and whether they are getting the right data to help the grid.  As a result, they’re already looking at a second generation of smart meters, despite the fact that the current meters were meant to have a 15 year life.  It means that even Spain’s low-cost smart metering deployment could end up proving to be a very expensive experiment.

Coming back to the question of how important a smart meter is, there’s an important fact that people need to grasp.  If all it does is send your energy usage to your supplier, it’s not very important.  If it goes wrong the energy supplier can go back to estimating bills or ask you to send in readings.  But that isn’t all a GB smart meter does.  Our smart meters have something else in them.  It’s an OFF switch, which can disconnect your gas or electricity.  That’s a little extra which the utilities added to save them the inconvenience of sending someone out to connect or disconnect you.  It’s so much easier if someone in their call centre, somewhere in the world, can just press a button.

At this point I need to do something I rarely do, which is to quote Josef Stalin, who is credited with saying that “A single death is a tragedy; a million deaths is a statistic”.  Once a smart meter has an OFF switch you need to turn that around: “A single smart meter is irrelevant; a million smart meters is a tragedy”.  Because if a million smart meters were to disconnect their users in one go, we could probably say goodbye to our energy infrastructure.  I don’t believe that this concept has ever got through to those involved in the smart metering roll-out.  Whenever I’ve raised the point about the consequences of someone hacking smart meters as a composite grid component, the only response I’ve had is “why would anyone do that?”  Which is akin to wondering why anyone would fly a plane into a building.  But they probably wouldn’t contemplate that, as it’s not in their Health and Safety training.

What is worrying is that this might not need a hacker – it could happen by accident.  The most high profile example of this is the recent crashes of Boeing’s 737 Max aircraft.  It appears that both were caused by firmware not being able to cope with the edge case of failed sensor, which no-one had expected to happen.  After the first crash everyone tried to convince themselves it was not a problem.  It took a second crash to see the aircraft grounded.  The very obvious lesson is that when a safety critical system starts to go wrong, you need to take it seriously.

With smart meters I am concerned that the level of software process, testing and understanding is nowhere where it needs to be to ensure the safety of the grid.  Smart meters are treated as individual devices with no concept of the damage which could be done if millions of them malfunctioned.  Security discussion within the smart metering programme is largely confined to data privacy and tampering, as the industry is still fixated on consumer fraud.  It probably wouldn’t be difficult for a determined extremist with computing expertise to get employed by one of the meter manufacturers and sabotage them, but that’s probably fanciful.  (Although I would hope there is some vetting of those working on the firmware).  What is more likely is a simple mistake.  The critical thing is how to deal with any such mistakes when they surface.

Last month we learnt what that response is.  Many customers of Bulb energy reported that their smart meters (although they probably meant their In Home Displays) suddenly changed from displaying English to showing Welsh.  The correct response to that would be to stop any further installations whilst conducting an immediate software review.  Instead this was treated as a joke, with the supplier’s spokesman responding to the failures by quipping that “We think Welsh is a great language”.  They might have acted differently if it had changed to Chinese and displayed the message that “Our Government is listening to you”, but they’d probably have made a joke of that as well.

It illustrates how ill-equipped the industry is for the magnitude of what it is trying to do.  Fifty million smart meters with a remote disconnect capability are not just individual billing units, but together become critical national infrastructure.  As such, that needs to be recognised at the design stage, through software verification and testing and monitoring of their performance.  Any unexpected behaviour needs to be flagged up as a cause of concern, with processes in place to determine what has gone wrong.  It will probably not be malicious; rather than being a Black Mirror plot, the cause is more likely to be a simple programming mistake that testing didn’t pick up, not least because of the insane pressure on developers and testers to get meters out of the door and shipped.  But that doesn’t mean we can laugh it off as a funny story for a quiet news day.  Each mistake like that has the potential to cripple the grid.

What is sad is that we have opened up this Pandora’s Box at immense expense, without including the features which would make it useful.  Instead we have exposed the grid to all of the risks and obtained virtually none of the benefits.  If the worst should happen, is there a contingency plan to reconnect a million or more homes which have lost power?  Our utilities are good at individual interventions like fallen powerlines and other utilities cutting through cables, but how would they cope with the need to replace or reset a million meters?  Have they even thought about that scenario?

It could happen.  Bulb have rather sheepishly reported that almost 30% of their smart meters are not working; another energy supplier told the press that 20% required a second visit from an installer.  Every time a smart meter exhibits a problem it should be logged centrally and investigated.  The saga with Whirlpool dryers shows the consequences of not doing so.  Once again, those are individual tragedies, not the infrastructure meltdown that could come from malfunctioning smart meters.

As more of these problems come to light, the industry needs to up its game in terms of security and take these reports seriously.  There is more to do than laugh as the world ends, whichever language you do it in.