An ‘epic fail’ for network resilience

I think we just failed an end-of-year exam on network resilience, Singapore.

My friend was returning home in a taxi last week when she texted me saying the driver had started the ride telling her that she could not pay by NETS (Network for Electronic Transfers) because there was “something wrong with the network”.

He proceeded to recount how he had spent the day at his company’s overcrowded service centre because he thought his taxi information update system was not working, only to be told after hours of waiting that there was “something wrong with the network”.

My friend kept up an amusing running commentary of this conversation to me via text. It was just a pity that it was a one-sided conversation — I did not get the messages till the next day. I guess there could have been “something wrong with the network”.

Last week’s fire at SingTel’s Bukit Panjang Internet exchange building damaged 149 fibre optic cables, which served OpenNet, StarHub, M1, SingTel, Nucleus One and a number of other operators.

Exchange points are an important part of network infrastructure; they are nodes where service providers (such as M1 and StarHub) interconnect and lease network lines from companies who provide the infrastructure that directly connects Singapore to the rest of the Internet.

As a former state-owned enterprise, SingTel owns most of Singapore’s fixed-line network infrastructure — while also being a service provider. Unless other telcos are willing to invest the time and money to install a parallel and competing network infrastructure to run their business, they will have to lease lines from SingTel — a direct competitor who has the incentive to preserve its monopoly and discourage new telco players.

SINGLE POINT OF FAILURE

Singapore’s network infrastructure set-up, having an incumbent own and lease infrastructure while offering telco and data services, led to a single point of failure situation during last week’s fire. The damage was done to a single exchange utility belonging to SingTel, but the impact was felt by customers of other telcos on leased lines.

This system failure escalated into a large-scale outage felt all over the island, no matter what service provider you were on. Household connections such as Internet, pay-TV services and digital voice calls were unavailable. Hypothetical fortunes were lost as betting services by Singapore Pools faced outages.

The breakdown went beyond telecommunication services — National Critical Information Infrastructure (CII) was affected: Banking and finance services on ATMs and in branches from UOB, OCBC, POSB and DBS were down; public transport was limited as taxi drivers were unable to connect to their information network, and telecommunication networks and health services were slowed, as a third of SingHealth polyclinics could not access patient records stored in the cloud.

The implications of the breakdown go further than consumers being unable to connect to their favourite social media website or place a bet.

One of the purported objectives surrounding the invention of the Internet was to allow continued information exchange even after critical communication infrastructure is destroyed in the event of a nuclear war.

If a single fire in a private company’s facility can cause five days’ service disruption in Singapore, it calls into question the resilience of the country’s CII. Further, it calls into question Singapore’s hard-won reputation as a hub of hubs: A central locale for the world to connect with, in order to connect to other hubs in the region.

A POLICY-REFINING FIRE?

Policy plans, such as the Infocomm Media Masterplan and the National Cyber Security Masterplan 2018, are being discussed to ready the country for the next technology wave. Recently, infocomm policies have been passed or updated: The Personal Data Protection Act was passed in November last year, and the Computer Misuse Act (1993) was updated in March this year.

G-Cloud (government cloud) infrastructure has been rolled out so that Singapore’s e-government services can leverage the power of cloud computing services and platforms. IPv6 adoption has been encouraged to support the coming Internet of Things, where everyday devices from phones to refrigerators have an IP address.

While much of the regulatory environment in Singapore is readying itself for Web 3.0, the Bukit Panjang fire has illuminated an important weakness in the country’s infrastructural set-up. At the time of writing, SingTel had not fully restored all connections damaged during the fire — StarHub and M1 still had circuits down, calling into question SingTel’s prioritisation in circuit restoration.

The question is: Will we repeat our mistakes? We may be about to build the same flaw into Singapore’s Next-Generation Nationwide Broadband Network, if SingTel, OpenNet and CityNet consolidate into a business trust that is owned by SingTel.

A coalition of telcos comprising Asia Pacific Carriers’ Coalition, M1, MyRepublic, Nucleus Connect, StarHub, SuperInternet and ViewQwest, have opposed the consolidation in a public consultation response paper (available online).

As the regulators review the fire and telcos compensate for service failures, we should use the lessons of the fire to inform the refinement of our regulatory environment.

ABOUT THE AUTHOR:

Lim May-Ann is the Research Director of TRPC, and Chair of Public Policy and Regulation, Asia Cloud Computing Association.