Beyond BPI’s glitch

Hopefully, by this time, BPI has fully restored its system. But there are so many lessons learned from what happened. I am sure the computer people of BPI and other banks as well, must have realized some vulnerability over what they call a “black swan” event. It could have been any other bank. 

The top management of banks, specially the major banks, should now realize the need to be ready with a crisis plan for events of this nature. BPI was grasping at straws, limiting themselves to issuing “advisories” probably written by their lawyers which failed to reassure their depositors. If they did their risk assessment exercise well, they would have had a plan to deal with a crisis of this nature.

A Google search tells me that bank computer glitches aren’t “black swans.” Bank computer failures are happening more often than we realize or are being told.

 In October 2015, The Guardian reports that Barclays had a glitch which they said was an internal one and was not caused by someone hacking into its computers. The head of the British Bankers’ Association was quoted by The Guardian saying that antiquated computer systems of banks are creaking under the strain of customers moving to online and mobile banking.

On April 28, 2017, itv.com reports that the customers of British bank Natwest saw their money vanish “into thin air” in a banking glitch. The glitch happened on a payday. One Twitter user wrote: “When you transfer money from a #Natwest account to another and it just disappears into thin air.”

A BBC news story quotes the strategy chief at a New York-based software analysis firm saying that the core of the problem is that the business software used by the institutions has become horrifically complex. He says developers are good at building new functions, but bad at ensuring nothing goes wrong when the new software is added to the existing mix.

“Business software is becoming increasingly complex, composed of sub-systems written in different programming languages, on different machines by disparate teams.

“This means no single person, or even group of people, can ever fully understand the structure under the key business transactions in an enterprise. Testing alone is no longer a viable option to ensure dependable systems.”

The other problem the expert told BBC is that there’s been massive underinvestment in technology in banks - “You hear stories of Cobol programmers being dug up and brought back from retirement after 20 years.”

BBC noted that this observation is a reference to the fact many banking applications are still based on the Common Business-Oriented Language. The code dates back to 1959 and is unfamiliar to many younger developers.

Mon Jocson, BPI’s EVP in charge of IT, assured me that BPI is adequately investing in new technology. This was confirmed to me by BSP’s incoming governor Nestor Espenilla.

“Ironically, BPI is one those most responsive to BSP requirements to put more resources into IT risk management. But in this case, human error caused the problem notwithstanding their better technology. It suggests need for further deepening training and expertise as well as more rigorous implementation of internal controls.”

Before social media, banks just quietly fixed their computer problems without the glare of attention BPI had. BSP must require the banks to have a risk mitigation plan and a crisis management plan that’s constantly updated.

BPI’s handling of last week’s “black swan” was so clumsy it isn’t worthy of its reputation as one of our top banks. A top priority should have been to re-establish trust that depositors lost with the first news of the glitch.

Other than the bank president giving reassurance over a cable channel with limited reach, BPI should have had someone who understands the technical nature of the problem explain the situation in layman’s terms.

Their SVP for marketing came from a consumer goods background and she is probably just getting familiar with the banking industry and was obviously not familiar with the IT problem that was the core of the crisis. I think she made things worse, even for me who had already understood and accepted the initial explanation.

I started to doubt their initial story after I watched her interview. She kept on saying depositors should not use outrageous sums of money mistakenly posted in their accounts.

But for that mistake to happen, that would mean it is not just a double posting problem as official BPI statements claimed.  Mon Jocson, the senior BPI official whose responsibility covers IT, should have explained instead.

Mon is no pushover. A UP engineering graduate, Mon was country manager of IBM Philippines and has worked through different positions including information systems manager, systems engineering manager and manager of quality for IBM. He has led IBM’s Applications/ Systems Integration business in ASEAN and South Asia and was based in Singapore as Asia/Pacific VP and GM for strategic outsourcing just before he joined BPI. He specialized in systems integration for banks.

I had a chat with him last Monday and he explained that it was a case of an overeager systems staffer who was cleared to work in their IT system’s heart of operations. She wanted to introduce some changes to improve the system, didn’t get prior clearance because she thought it was simple. She then picked up a wrong set of files.  

By daybreak, her mistake was propagated into the entire system including the back up files. First to notice the problem were call center employees getting off their overnight shift. They voiced alarm over social media, and all hell broke loose.

Correcting the error took time because of the volume of files affected. They removed the control gates that could have limited the damage last December to allow for heavy Christmas traffic and had not put it back.

Mon said one of the first things he did was to check with IBM, the entity handling their data security, to make sure it is not a hack. Assured that it wasn’t, they focused their efforts on fixing the internal error.

Inasmuch as all the big banks are using systems similar to BPI’s and following similar protocols as well, BSP must conduct a thorough review of everything. And by everything, I also mean checking if the banks have a credible crisis communication plan.

Indeed, I think the crisis mitigation plans should include naming a bank spokesman who is knowledgeable about IT and bank operations. The SVP for marketing they tasked to talk to media was essentially a non-banker and non-IT person. 

No wonder she was reduced to talking about trust when trust was a scarce commodity for BPI at that moment. She kept on saying BPI was over a hundred years old as if it mattered at that time.

A big problem for BPI over those two days of outage was social media. The malicious memes and fake news posted and shared extensively should have been actively addressed. But the bank’s corp comm group is likely too low in the totem pole to get top management respect. They were obviously too intimidated and inexperienced to handle the big problem.

BPI’s experience is an excellent case study of what not to do and what ought to have been done. The bankers, the bank regulators and corporate communicators of other banks should carefully study the lessons learned and be prepared. 

Having worked in two banks myself, this case is something we would have loved to learn from in the old Bank Marketing Association. How BPI tries to win back their depositors’ trust will be interesting.

Boo Chanco’s e-mail address is [email protected]. Follow him on Twitter @boochanco.