SingHealth data breach: Sleuthing by IHiS staff helped authorities, hackers used PM Lee’s IC number

SINGAPORE – He noticed unusual database activity on July 4 that did not make sense to him.

Instead of letting it go after the query stopped running, an assistant lead analyst at the Integrated Health Information Systems (IHiS) – the Ministry of Health's IT arm – decided to probe further.

Mr Chai Sze Chun was the first IHiS staff member who tried to ascertain if there was malicious intent behind the query, or request for information.

His sleuthing helped the authorities in uncovering the massive data breach at public healthcare cluster SingHealth, as more details emerged from the second day of public hearings by the Committee of Inquiry (COI) looking into Singapore's worst cyber attack to date.

And in the aftermath, Mr Chai produced a log of queries, one of which showed the hacker making a direct query for data using the identity card number of Prime Minister Lee Hsien Loong. The hacker made a direct query on two others, but they were not Very Important Persons, Mr Chai said in his evidence to the COI.

Other queries related to demographic data of patients and the medication that was dispensed.

Between June 27 and July 4, sophisticated hackers stole the personal data of 1.5 million SingHealth patients and the outpatient medication data of 160,000 of them, including PM Lee.

Mr Chai's job was mainly to support end-users who had trouble accessing the system either directly or via bridging systems like Citrix servers.

"Mr Chai was an IHiS officer whose actual job was not cybersecurity management but ensuring operational efficiency. Nevertheless, when faced with unusual circumstances, he was alert and he showed initiative when investigating into the security incident," said Solicitor-General Kwek Mean Luck, who is leading evidence in the inquiry.

Around lunchtime on July 4, Mr Chai received text alerts and went on to check if there were persistent performance issues with the Sunrise Clinical Manager database server.

He noticed a query that had been running for a while. When he saw later that it was no longer running, he investigated and tried to find the user-ID of the person logging in to the Citrix server from a workstation.

Efforts to trace the user were futile, and he asked a colleague for the logs of users who had logged in to the Citrix server. He found no record of the particular workstation having logged in that day.

Four possible reasons occurred to Mr Chai, two of which meant an account had been potentially misused.

He informed other colleagues including Ms Katherine Tan, the Sunrise Clinical Manager database administrator, via emails and continued seeking answers on the unusual query.

Ms Tan informed him of similar queries she had observed, and they were unsure who was running the queries and decided to terminate the processes to see if any user or colleague impacted would call them.

They did not receive any calls.

Ms Tan testified last Friday that she went home on July 4 and developed a script to stop more unusual activity, completing the task at about midnight.

Before he went home that day, Mr Chai tried contacting others to try to ascertain the exact location of a workstation in question, and also set up a Whatsapp chat group for "quicker communication with (his) colleagues on this matter".

The next day, he decided to determine the earliest date on which the queries had been run on the database, involving the combination of the particular account and programmes (which were redacted in documents made available to the media). He found the queries had begun on June 27 and "there had been many such queries" until July 4.

He informed his colleagues. Mr Chai said that after reporting his findings, he understood that the Security Management Department was investigating the incident.

About five days later on July 10, IHiS set up a "war room" to trawl the patient database, to look for all failed log-in attempts.

Mr Chai was one of three IHiS staff members who testified to the four-member COI on Monday. The others were his supervisor Steven Kuah, assistant director of the Production Enhancement Team, and Mr Chan Chee Choong, manager of the SingHealth Active Directories, who is in charge of password policy settings.

Like other IHiS employees who testified last Friday, Mr Kuah and Mr Chai said they were not aware of the formal security incident reporting framework at the organisation.

Both said they do not remember receiving training on this framework.

Before July 4, Mr Chai said he was not aware of any specific requirements as to whom and how to report an IT security incident, or that specific timelines existed for the reporting of such incidents.

The public hearings are set to continue on Tuesday and Wednesday (Sept 25 and 26).