SingTel contingency plan questioned as fire repairs continue

SINGAPORE — As SingTel struggled to fully restore all services that were affected by Wednesday’s unprecedented disruption, questions were being raised yesterday by various quarters — including businesses and industry observers — about the contingency plans in place for business continuity as the island gets wired up.

The disruption caused by a fire to SingTel’s Bukit Panjang facility — which hosts systems and equipment such as telephone switches and cables for telecommunication services — disrupted a range of services across the country and affected tens of thousands of people.

ATMs and other banking services were down, as well as some services offered by other telcos. Schools and Government agencies were also affected by the disruption which entered a second day yesterday, despite SingTel earlier pledging that all services would resume by 6am.

In the end, SingTel managed to restore all corporate services by 7pm and it said it expected the remaining services to be repaired fully by 7am today.

At a hastily convened press conference, SingTel Consumer Singapore CEO Yuen Kuan Moon said the telco discovered that the damage was more severe than initially thought and its engineers had to work in “extremely difficult and challenging situation”.

SingTel engineers, armed with their equipment, had to work in a fibre chamber — there are three chambers in all at the facility — that measured just 40m by 5m, said Mr Yuen. “(It) is actually quite challenging ... a lot of people need to be working on it in a cautious manner to make sure that safety is of utmost importance,” he added.

On Wednesday, SingTel initially said that 33 cables were damaged. However, the actual number was far larger: In total, 149 cables were damaged. Other factors also made repairs challenging, said Mr Yuen. For instance, the fibre cables were colour coded but the fire charred the damaged cables.

SINGTEL TO SET UP BOARD OF INQUIRY

The telephone exchange in Bukit Panjang is one of nine located islandwide. On the security measures in place, a SingTel spokesman said the gazetted facility is a key installation building protected by armed security personnel. Contractors need to be authorised before they can enter the facility.

The telco’s immediate priority was to repair all services, and whether it intends to file any police report will depend on the outcome of ongoing investigation, the spokesman said.

Mr Yuen said that a board of inquiry will be set up to look at how to better protect SingTel’s network and its distribution of services.

It is understood that the Infocomm Development Authority (IDA) leaves network design up to the individual telcos, with the condition that it has to be resilient in all aspects.

Incidentally, IDA had announced last week that it would be implementing a new audit framework to regularly review the resiliency of mobile networks and it would also be working with mobile services operators to enhance resiliency and minimise possible disruptions to consumers and businesses.

In a review of the three telcos - SingTel, StarHub and M1 - IDA had noted that their network resilience and redundancy design, as well as facilities and infrastructure support system generally met international standards, and in some areas were among the best in the industry.

However, it flagged business continuity planning and network design, as areas that the telcos could do better in.

Following Wednesday’s massive disruption, some observers TODAY spoke to felt that IDA should increase its oversight of telcos to ensure business continuity planning and network resiliency.

Chua Chu Kang GRC Member of Parliament Zaqy Mohamed, who heads the Government Parliamentary Committee for Communications and Information, said: “It comes down to how IDA plans the whole telecoms network resiliency... It is time to re-look communications resiliency in Singapore, and continuity plans – how to react if certain key nodes go down.”

Former journalist and technology blogger Alfred Siew said this incident should be a red flag for the IDA to be more thorough with its new audit framework, and extend this to fixed line networks. “IDA should take into account this incident when it finalises the audit framework,” he said.

Some businesses, especially those which are more reliant on cable services, also called for more to be done. Mr Travis Ho, a director of game development company Touch Dimensions, noted: “Online gaming companies would be affected (in a prolonged disruption) as so many players go online at any one time to play, unless their servers are hosted overseas.”

'A LEARNING POINT', SAYS YAACOB

At the press conference, Mr Yuen stressed that systems were in in place to ensure business continuity, by replicating equipment so that services could be diverted if necessary. “Physical location diversity’ is achieved by having nine telephone exchanges across the isiand. And by having separate fibre chambers at each facility, there is “additional diversity and redundancy”, he said.

He added: “In this particular instance, you are talking about physical damage... the fibres being burnt... You need to have physical work, people going down to the ground to fuse the fibre back together. So there are certain things we cannot replicate very quickly.”

He reiterated that SingTel adopted “best practices of fire prevention in all the exchanges”. “I cannot answer the questions of whether... what is required, what is not required. Only (after) full investigation when the results come out, then we will be able to answer some of these questions,” he said.

Speaking to reporters after a briefing by SingTel and a site visit to the Bukit Panjang facility, Minister for Communications and Information Yaacob Ibrahim also stressed that pending the investigation, it was premature to say whether security and fire prevention measures were adequate.

He added that he was “quite happy” with the way SingTel handled the disruption. Adding that the telco mobilised all its engineers - numbering more than 100, he said: “They’ve used social media to communicate with their customers, which I think is a novel approach, in order to ensure that customers understand; and they’ve been transparent from day one, which I think is very good.”

On whether there was sufficient contingency measures to ensure business continuity, Dr Yaacob said this was “an area which IDA has always been concerned (with), long before anything has happened”. Adding that systems “are becoming very complex”, he said: “Every incident will be a learning point for us. Let’s wait until we finish the investigation to find out what more that we can do to strengthen our infrastructure.”