Apple, Samsung and Volkswagen. How firmware can damage your business.
- Published
- in Design
It should have been a good Christmas for Apple. Millions of happy Apple fans were likely to unwrap one of their products on Christmas Day. But just a week before, Apple got a present it really didn’t want. The news broke that they had been releasing updates which slowed down the performance of older phones.
The reason for doing this is that as lithium batteries age, their performance gets worse. If you keep on putting the same demands on them, there is a double risk – they may degrade faster and need to be replaced, or in an extreme case, they could fail, possibly disastrously. So, there is a definite logic in trying to limit those demands to keep the user physically safe.
However, it’s a difficult concept to sell. Consider if an automotive manufacturer were to do the same thing with your electric car. If you bought the car on the basis that it had a top speed of 80 mph and a range of 200 miles, you’d probably be rather irate if, twelve months later, you discovered that they’d decided to restrict the top speed to 35mph, in order to ensure that the range didn’t fall below 200 miles. But that’s what the headline claims against Apple are implying – that unbeknownst to the users, software updates are deliberately throttling back the phone’s performance. The electric car example above is not a valid comparison, but to understand why requires a level of technical knowledge that few journalists or lawyers possess. They’d rather cast Apple as the villain, turning this into an Applegate conspiracy. Viva fake news.
I’m as sure as I can be that Apple made these changes in good faith. Their only fault was that they didn’t communicate the reasons for them with users. But that brings us to the centre of the issue; the role of firmware in the smart products we use today. Few people have heard of firmware or know what it is. It’s the software which is embedded in the microprocessors of the devices we use, controlling their most basic functionality. It’s put there by the manufacturer and can’t be altered by apps or anything a user can add to a device. As such it controls the core aspects of how each device works. It used to be loaded once and for all at the point of manufacture, but with modern, connected devices, it’s often possible for a manufacturer to update it. In the case of cars, that used to happen at a service. In smartphones and other connected devices, an update can be loaded remotely by the manufacturer, with the user largely unaware of what that means. At best, they’re told that the device needs updating, but rarely informed about what that means for its future performance.
As users clamour for ever more features on their devices, the role of the firmware becomes ever more critical. Devices like smartphones and cars are being designed at the bleeding edge of what is possible, where the constraints usually come from two different and often conflicting directions. The first are regulations – external requirements that a product needs to meet, such as diesel emissions, safety requirements or wireless power limitations to prevent interference. The second are technology restrictions, which for most mobile products come down to battery life and power management.
Batteries are the Achilles’ heel of mobile products. Whilst Moore’s law has allowed us to design more and more powerful microprocessors, and other technology advances have given us larger and more brilliant screens, battery technology has largely hit a road-block. The Lithium Ion batteries which are used in smartphones have not made a major leap in power capacity or lifetime over the last decade. In order to accommodate the extra features which are poured into smartphones, consuming ever more power, considerable ingenuity has gone into pushing the performance of these batteries right to the limit. Go into any phone company and one of the largest engineering teams will be the one working on power management, striving to extract the last few joules of energy from the lithium-ion battery.
The result of this innovation is that the industry is now finely balanced on the edge. Take a little too much power and lithium ion batteries start exploding or catching fire. That’s what happened to Samsung in September 2016 with its Galaxy Note 7 phone. A new battery design took it just over the edge, resulting in phones catching fire, and an eventual recall of the entire production run. It was interesting to note the lack of any criticism from Samsung’s rivals. That’s almost certainly because all of them were aware that it could so easily have been them. The line between a high performing smartphone and an exploding smartphone may be just a few lines in the power management firmware.
Understanding the importance of managing firmware and understanding the consequences of a failure to manage it has only recently appeared in the radar of senior management. It hit the headlines with the Volkswagen emissions scandal, where it emerged that vehicles had engine management firmware which detected the characteristic driving conditions of a regulatory test and adjusted the engine performance to ensure that it complied with the emissions requirements. As soon as it was back on the road, the engine management parameters were changed to give the performance that drivers wanted, but which exceeded the allowable levels of emissions. I’m still not sure that the full details of how this evolved within Volkswagen are known. It’s certainly been the case that firmware engineers attend most regulatory tests to tweak software if that’s needed to ensure the test is passed. That’s normal practice, but those changes are then incorporated in production code. What was different with Volkswagen is that someone deliberately added a new program which detected the regulatory text environment and selected one set of engine management parameters for the test and a different one for everyday driving. That is clear fraud. It’s not the type of thing firmware engineers are likely to do, suggesting input and knowledge from more senior people in the design chain.
That gets us back to the conspiracy theories, which is why Apple’s firmware updates could turn into Applegate, as there is the suggestion that Apple may have felt that degrading performance of phones might help persuade users to upgrade to a new model. There is no evidence for that, but lack of evidence never stopped an avaricious lawyer filing a class action. At the time when it appears that Apple first started working on these performance-limiting firmware updates, two things were happening in the market:
- Samsung were recalling their Galaxy Note 7s because of the battery problem, and
- Industry analysts were noticing that users were delaying upgrading to newer smartphones.
Put those together and you can imagine the following conversation in a meeting:
Firmware Engineer: “We think we need to put out a firmware update that reduces the processing load on batteries, so that they remain safe. We probably need to throttle things back by around 30%. We don’t think most users would notice that.”
Engineering Manager: “Given the Samsung experience, is that enough? They had to recall two and a half million phones, but at least it happened at the launch. If we had a recall after a year it would mean three hundred million phones coming back. Should we play safe and go for a 40% reduction. Would users notice that?”
Product Marketing Manager: “Hang on – how far would we need to go to make sure they noticed a reduction? That could be a way to get them upgrade a little earlier, so we kill two birds with one stone.”
One shouldn’t castigate the marketing manager for having that thought – that’s the sort of connection they’re paid to make. If they don’t make it, your company needs a better product marketing manager.
It’s inconceivable that the conversation was as simple or as blatant as this, but it almost certainly happened, probably in multiple meetings and it’s equally probable that it got passed up the management chain to someone more senior to make a decision. I very much doubt that the decision to go ahead was based on anything other than the safety of its users. Apple made the correct call, which was to apply an upgrade which throttled the performance of phones as their batteries aged. But they made one wrong decision, which was not to inform their users. That’s given free rein to the conspiracy theorists and lawyers who want to get their hands on as much of Apple’s cash pile as they possibly can.
The throttling back of performance does raise one important question, which is largely being ignored – just what performance can a user expect when they buy a product like this? Should the sales proposal tell us about the performance we will get on day one, or should it be based on the performance at a later point, say halfway through the anticipated product life? Most people expect performance to decrease as a product “wears out”, although that’s based on observations which traditionally are informed by mechanical wear of components. The fact that the diminished performance could be programmed by the manufacturer, and that it could be applied after we’ve bought and are using the product are alien and smack of the worst aspects of planned obsolescence. In the current case and in many others, I don’t think that’s the case. Rather, it’s the prudent management of the ageing of critical components. But most users are unlikely to see it that way. Hence the need to be upfront and tell them before they buy the product.
Apple seem to have grasped the enormity of their mistake and have offered a reduced price battery replacement for users of older, affected phones. Whether that will take the wind out of the legal action is unclear. Once lawyers sniff the possibility of easy money, they tend to become somewhat single-minded. But there could be one positive outcome from this episode. Up until now, firmware hasn’t really featured in the consciousness of most senior management teams. It’s just been one of those annoying things which always takes too long to complete and delays product launches. The Volkswagen debacle has shown what goes wrong when you let it be subverted (although I still wonder how much senior management was actually aware of). Samsung has illustrated how dangerous it is to get too close to the edge and now Apple is facing the consequences of assuming it’s too esoteric to tell their users about.
Apple are not alone. Users are becoming aware of the verb ”bricked” – an engineering term to denote an event that stops a product working. In the past twelve months, multiple companies have deployed firmware updates which have bricked devices including TVs, HVAC systems and door locks. It’s time for companies to understand that firmware is not just a component, but increasingly the item which keeps them on the right side of the bleeding edge of technology. It needs more understanding all of the way up the management chain, as getting it wrong can be disastrous for a company’s reputation. As I said before, I suspect Apple did everything right, other than share the information about what they were doing with their users. If the wider industry learns from their pain, it should ensure a better product experience for consumers as more and more of our devices become connected.