SINGAPORE — Following a recent technical glitch that sparked hours-long delays and snaking queues at Singapore's checkpoints, the Immigration and Checkpoints Authority (ICA) is reviewing its processes when it comes to upgrades to its immigration clearance system.

Around 85,000 travellers were affected — including 21 passengers who missed their flights — as a result of the close to 4.5-hour problem on March 31, which started at around 10.40am.

Associate Professor Muhammad Faishal Ibrahim, Minister of State for Home Affairs and National Development, said this in Parliament on Friday (April 21) when he gave details on what had caused the glitch, in response to several parliamentary questions from Members of Parliament. 

WHAT HAPPENED

On the morning of the incident, ICA was conducting a pre-scheduled trial that was needed for an upgrade of a system, which facilitates automated immigration clearance using travellers’ biometrics.

Unfortunately, the trial caused a severe system overload, which affected the automated clearance lanes at all the departure halls in Changi Airport and some automated lanes at the Woodlands and Tuas land checkpoints.

The rest of the automated lanes, including those at the sea checkpoints, remained operational, Assoc Prof Faishal said.

During the disruption, about 55,000 travellers passed through the two land checkpoints, while about 30,000 passengers on 113 departure flights and 111 arrival flights cleared immigration at Changi Airport.

He did not give details on how many travellers would have cleared immigration under normal circumstances. 

Before the incident, ICA had been upgrading its systems “progressively and cautiously”, with 10 out of 12 systems enhanced, he added.

For the system that was being upgraded on March 31 — called the multi-modal biometrics system (MMBS) — the upgrade involved replicating the large biometric database to an upgraded system.

“This has to be done continuously over a few days, and the MMBS has to remain operational during this period,” he said.

Any system upgrade typically undergoes a trial test first at different times of the day, before proceeding with the actual upgrade.

Before the upgrade on March 31, a trial was successfully conducted on March 15 from 1.30am to 3.30am.

Assoc Prof Faishal said that systems vendors were on standby on-site to recover the system within 30 minutes if the upgrade did not go well.

This time round, it caused the storage systems to overload at about 10.40am and the process was aborted immediately.

“However, the extent of the overload was much more severe than anticipated, and the vendors who were on-site had to work with their global support team to diagnose and reboot the servers,” he added. 

IMPACT OF DISRUPTION

Assoc Prof Faishal said that ICA immediately activated its business continuity plan once the system went down.

Off-duty officers were recalled to help man manual immigration counters and perform crowd control.

Across all the checkpoints, the failover process kicked in and all the manual counters, as well as certain automated lanes, switched to backup systems.

However, not all the automated lanes have this failover capability, because different models were procured over the years.

At the land checkpoints, travellers experienced delays of up to 30 minutes at the onset of the incident, Assoc Prof Faishal said.

Those travelling by car at Woodlands Checkpoint faced another hour as the car arrival zone had to be converted to clear motorbikes manually.

In the meantime, immigration clearance at the Changi Airport was “significantly impacted”, he added.

Travellers were re-directed to manual counters for immigration clearance, which saw ICA recalling off-duty officers and deploying administrative staff members and retaining the officers from the outgoing shift.

Staff members from Changi Airport were deployed to identify travellers whose flights were departing soon so that their departure clearance could be prioritised.

Despite the measures taken by ICA and Changi Airport, 21 passengers missed their flights at Terminal 4, Assoc Prof Faishal said.

These travellers were later offered to re-book on alternative flights within a week at no extra cost.

Assoc Prof Faishal said that even with the implementation of business continuity plans, degradation in service standards was “inevitable” given the volume of travellers especially at the onset of the incident.

“We apologise to affected travellers for the inconvenience caused and thank them for their understanding.” 

PREVENTING FUTURE DISRUPTIONS

In light of the incident, ICA is reviewing the approach to the upgrade.

"The alternative approach that ICA was initially offered by the vendors was not tenable. It involved upgrading the system directly without replicating the database, but may require five days of system downtime," Assoc Prof Faishal said.

Going forward, ICA is in the process of rolling out a multi-year plan, known as the New Clearance Concept.

This will transform immigration clearance across all the checkpoints by providing more seamless clearance, he said.

It will also help ICA, with limited manpower resources, cope with the ever-rising traveller volume.

Under the new concept, all manual counters and automated lanes will be replaced in phases with the next-generation gates.

This will allow travellers to continue using automated clearance even when the MMBS is down.

In the meantime, the existing automated lanes that have yet to be replaced by the new gates will be upgraded in the second half of this year.

Once done, Singapore will become one of the first countries in the world to implement the New Clearance Concept-type capabilities, Assoc Prof Faishal said.

That said, disruptions will still happen now and then, he added.

“When they do, we will make sure we have robust contingency plans in place, and try to recover as fast as possible. I hope for the public and Members of Parliament’s understanding."

The recovery process took about 4.5 hours and was back up around 3pm.