O2 up and running but how much damage has been done to them and Ericsson?

With O2’s UK network back up and running, the 32 million Brits who depend on it have been returned to the digital era, but you have to wonder how big the fallout from this disaster will be.

With 3G data services restored late Thursday evening, and 4G getting the green-light early Friday morning, a stressful period comes to a close. Now the more difficult questions need to be asked to understand why this happened, why O2 is rumoured to have had to cancel its Christmas party last night and what the consequences of the chaos will actually be.

“We can now report that our 4G network has been restored,” said an O2 announcement. “Our technical teams will continue to monitor service performance closely over the next few days to ensure we remain stable. A review will be carried out with Ericsson to understand fully what happened. We’d like to thank our customers for their patience during the loss of service on Thursday 6 December and we’re sorry for any impact the issue may have caused.”

In fairness to O2, the simple thing to do here would have been to shift all of the bad press and finger pointing towards the root cause of the problem, Ericsson, but it has managed the saga as well as could be expected.

And while it might have taken a couple of hours for the Swedes to come clean, they finally did, as you can see below:

“The faulty software that has caused these issues is being decommissioned and we apologize not only to our customers but also to their customers,” said Börje Ekholm, Ericsson CEO. “We work hard to ensure that our customers can limit the impact and restore their services as soon as possible.”

While Ericsson is continuing to do root cause analysis on the fault, the issue has been narrowed down to two specific software versions of the SGSN–MME (Serving GPRS Support Node – Mobility Management Entity). These nodes in the core of O2’s network caused the calamity, though it was not alone.

Softbank experienced the same issues over in Japan, while there are rumours several telcos in Asia also had network outages. Ericsson has confirmed it impacted other customers, but it is not its place to name said customers; O2 and Softbank made their own announcements, so it is up to the operators themselves. Mobifone in Vietnam could well be one of these customers, with its own network shutting down at 11.30am local time.

Before we move on, it’s worth drawing attention to the graph below from web performance and security company Cloudflare, just for context.

As you can see the drop was incredibly pronounced, though the minor traffic which can be seen has been attributed to O2 customers who were roaming outside the UK. These customers do not seem to have been impacted by the data-doomsday.

One of the big questions which is now floating around concerns the fallout. With people and society on the whole relying so heavily on data networks, any fear of sub-par or non-existent performance will have a negative impact. Outages are something people should realistically expect to happen every now and then, but the severity and length of this one is certainly noteworthy.

“The disruption shows how much importance we place on a mobile device,” said telco and media analyst Paolo Pescatore. “Without connectivity, people are stranded, and businesses cannot compete. Furthermore, it underlines the need for continuous investment in the UK digital infrastructure, both fixed line and mobile networks to ensure growth and productivity.”

Just scrolling through Twitter, you can see how many people were impacted by the outage. This impact cannot just be restricted to personal activities, as there have also been several reports of sole-traders and critical services being unable to do their jobs. Whether we’re talking district nurses being unable to do house-calls because they are unable to rely on mapping apps or a plumber who can’t access his emails, the consequence is incredibly real and financial.

Earlier this year we had the chance to speak to various O2 executives, including CEO Mark Evans, over dinner, and the enterprise market was a segment targeted for growth at the firm. This is a lucrative market currently dominated by EE and Vodafone, though with this outage you have to wonder what the cost will be for O2. Joe Bloggs having to speak to someone on the bus is a minor inconvenience but shutting down a business which relies on mobile to work effectively is a completely different matter.

On one side of the coin, you have to feel a bit sorry for O2. This is a business which has been effectively shut down due to a fault from one of its network partners. Customers will not actually care what the problem is, they only deal with O2 therefore O2 should shoulder the blame. The negative impact on brand credibility and the company’s ability to offer basic services will certainly be questioned by some. Conversely, Ericsson’s share price has actually gone up a few percent since this news broke, perhaps reflecting the apparently swift resolution of the crisis.

However, you also have to wonder whether O2 is itself culpable. Should this be considered a due diligence or supply chain issue? As you can see from the tweet below, Heavy Reading’s Gabriel Brown doesn’t feel O2 is innocent through the saga:

And sticking with Twitter to finish, as you can imagine there were certainly a few people who had fun with the toil and torment of others. We’ve copied a couple of our favourite tweets from the last 24 hours below. Enjoy.