Protecting Key Pathways: The Payments System Keeps Getting Busier and More Complicated. CUs Must be Vigilant to Prevent or Fix Outages.
By Richard H. Gamble
Keeping the payments system running has always been so critical that elaborate arrangements of hardened perimeters, file back-ups, processing redundancies, frequent testing and many more such steps have been used to keep outages within acceptable ranges.
Today, as payment options proliferate, real-time transactions take over and sophisticated hacking becomes more threatening, alert credit unions continue to check their own shops and look for weaknesses in the overall payments infrastructure.
Operations planners must understand how all payment-related systems work and interact, suggests
April Vuylsteke, director of global financial services at Hyland Software (hyland.com), Westlake, Ohio.
Then they must identify where disruptions might occur, take steps to prevent them and plan to keep
critical services working when disruptions do occur. Test by simulating outages that trigger response plans, she recommends. And make sure vendors have protected and tested their roles, especially around payments; have agreements and service-level requirements in writing.
Sometimes high enough volumes can cause systems to choke temporarily, such as when COVID-19
mitigation payments were supposed to arrive in member checking accounts. Some CU online response sites got overwhelmed with members checking on those anticipated deposits.
Even this minor “outage” was felt in the call center, as members phoned in when digital systems
didn’t deliver. “We handled 20,000 calls on behalf of CUs one day,” recalls Tammy Snyder, SVP/contact center services at CUES Supplier member CO-OP Financial Services (co-opfs.org), Rancho
Cucamonga, California.
Whenever a system is down, members will expect answers, Snyder says, and the best thing a CU can do, whenever possible, is answer those questions before they reach a live rep. Post a notice on the mobile banking site, which is the first place most members look, and a recorded message on the call center’s interactive voice response. “Let them know that you are aware of the problem and tell them when the system should be back up.” But send the calls through, she says, in cases where a rep could act on the member’s behalf.
Too Much Volume
One of the oldest causes of an outage is sheer volume overwhelming a system and causing it to freeze. When this is done deliberately, it’s called a denial-of-service attack. When accidental, it’s called poor planning, observes Stuart Bain, SVP/product management at Alacriti (alacriti.com), based in Piscataway, New Jersey. The widely reported Wells Fargo outage (tinyurl.com/ermmfw7j) on March, 17, 2021, was caused by a huge wave of customers all trying to sign in at once to see if stimulus payments had been posted to their bank accounts.
“That outage could have been prevented,” he says, with better planning and communication—anticipating that those calls would be made and telling members how and when the payment would arrive before they asked.
When an outage occurs, waiting may be the best choice, as most are fixed sooner than a workaround could be built. Having redundant systems ready to go is often too expensive, Bain adds.
A form of planned waiting is widely practiced by airlines, notes payments consultant Richard Crone, founder/CEO of Crone Consulting LLC (croneconsulting.com), San Francisco. Historically, flyers paid cash for a drink or a meal. Now airlines only take cards,but since the card terminals don’t communicate from 50,000 feet, the terminals have an important feature—store and forward. The terminals capture the transaction data and store it until the plane lands, then forward it to a processor, he explains.
“The terminals work offline and synchronize with the processor when the plane lands.” Similarly, an alert CU team wants their card processors to have store-and-forward capability on the ground so card payments can continue to be accepted during an outage and then clear quickly when it’s over, he recommends.
Routing Solutions
When waiting won’t work, stalled payments need to find an alternate path. Sometimes one party can cover for another, known as “stand-in processing,” reports Sabeh Samaha, founder/CEO of Samaha & Associates (ssamaha.com), Miami Beach. If a CU’s link in the chain goes down, the next link, say a card payments processor, could step in and allow member payments to advance outside the normal routine, essentially trusting that the CU will provide the missing data and settle the payment once its system is back up.
The credit union’s core remains “the system of truth,” Samaha says, the one with the records of members, accounts and transactions. Unless the processor has a current positive balance file, it is allowing payments to proceed based on trust that the funds are there. For protection, a stand-in processing provider usually limits the number and dollar amounts of transactions it will stand in for, he explains.
An upstream processor is just one possible rerouting solution. For example, a CU could accommodate members during a card payment outage, Bain explains, by routing or having the card processor route the payment to the ACH (nacha.org/content/ach-network) for batch processing and overnight settlement. The CU would need to ask affected members for their account details so the CU or processor could send an ACH payment, because card payments cannot be converted to ACH automatically.
Open application programming interfaces certainly are helpful in these situations, Bain adds, because they essentially unlock the doors to intelligent routing. “Open standards allow alternative systems to be tapped,” he explains. “It allows payments to flow regardless of the network, which makes it easier to switch networks if one is down.”
Switching can be done at the processor level without involving the networks. But you can’t reroute payments without the relevant data. A hiatus in payment movement is inconvenient; losing data can be catastrophic, he notes.
Even when all payment pathways are working, systematic intelligent routing through open systems allows payments to take the most appropriate paths to fulfill the instructions, Bain continues. “Members want to see their wishes carried out. They usually don’t care how that happens, which leaves the credit union free to find the best route.”
But all steps in the process need to work perfectly almost all the time, because multiple handoffs provide many points of potential failure where outages can occur. “The need for robust back-up and redundancy is magnified,” Crone notes. The result, he says, is an industry “triple-nine reliability standard—verification that the systems will be functioning 99.9% of the time,” theoretically making unplanned outages extremely rare.
Cloud Exposures, Protections
Payments highways often go through the cloud or over the internet. “That entails more points of vulnerability,” Crone notes, “but also more rerouting possibilities. Some see the cloud as a single point of potential failure, but it’s not.”
The cloud helps prevent outages, Bain agrees. A CU that runs its core system on two physical servers could suffer an outage if the primary server fails, leaving a single back-up. A cloud-based setup will have many virtual servers linked to support core operations and payments.That makes the cloud better able to respond to demand and makes it possible to take some systems out of service for updates without members even noticing, he explains. A core conversion or update may mean shutting down the core temporarily if it’s running on in-house servers. “In the cloud, service can be continuous.”
With payment systems functioning along chains, Crone notes, CU vendor management programs need to make sure all payment processors they use have met security standards—that they have satisfied SSAE 18 audit requirements. CUs need to verify those audits and investigate any exceptions.
Credit union bill-pay programs deserve special attention. A lot of time-sensitive payments are embedded in them, Samaha points out. “Many members count on funds posting on paydays, with scheduled payments to follow quickly,” he says.
If bill-pay goes down due to a system conversion or repairs or unplanned disruption—perhaps at the bill-pay vendor—overdrafts or missed due dates could follow. Therefore, it’s critical that CUs alert members as quickly and effectively as possible if there is a glitch in their bill-pay services. Then members can consider alternatives, perhaps assisted by member service reps, like making a critical payment by phone using a card.
Bill-pay is vulnerable, Vuylsteke agrees. “Those services usually rely on the ACH or FedWire (frbservices.org/financial-services/wires) on the back end,” she says.“We saw an outage at the Fed in February 2021 and a problem with duplicate payments at Wells Fargo in 2018.” CUs that tailor the service for members could offer a secondary bill-pay option that uses a real-time payment rail and charges a small fee. Similarly, business members could be offered a real-time primary or back-up option for things like payroll and insurance payments, she suggests.
Including the Innovators
Recent payment alternatives like person-to-person networks and cryptocurrencies also add pathways that can be used to move payments past blockages.“One of the best ways to cope with payment systems outages is to have multiple, diversified ways to make payments, and the P2P networks are a reliable alternative,” Crone notes.
Thousands of CUs still don’t offer P2P payments like Zelle (zellepay.com) and Venmo (venmo.com) through their mobile banking apps, and that’s a mistake, he says. “Venmo, Zelle, PayPal (paypal.com) and Square (squareup.com) are now separate and distinct payment systems that give CUs and their members more options when traditional payment systems are down.”
P2P payment systems may ride their own rails once accounts are funded, Bain counters, but unless a person keeps a balance in a Zelle, Venmo or PayPal account, the process likely would start with a charge to a credit card or debit from a bank account, which limits their usefulness as a true alternative.
Cryptocurrencies like Bitcoin are another emerging payments system to consider. Blockchain is a set of rails that is unaffected by what happens to card networks and bill-pay clearings, Crone says.
CUs haven’t offered cryptocurrencies yet, but they soon could because FIS (fisglobal.com), a major core processor for CUs, has announced an arrangement with NYDIG (nydig.com) that would accommodate Bitcoin holdings within its users’ mobile banking apps. This would make it possible to fund a crypto wallet from a mobile banking wallet.
How Strong is Strong Enough?
Payment system outages happen, but payments continue to flow because the systems are resilient, says Arvind Sharma, chief digital and payments officer of $23 billion Central 1 Credit Union (central1.com) in Vancouver, British Columbia.
“We’re a processor. Once in a while, something happens, and then people work through the night to get it fixed,” he explains. “Usually, individuals using those systems never notice.”
Blips do occur. A member’s connection to a CU’s mobile banking service might fail, or a CU might shut down a system for maintenance for a night after notifying members, or a member’s card payment might not go through due to security triggers, but the systems are safe, he says.
But challenges are real. As the reliable backbone of the U.S. interbank payments system—FedWire and ACH—move to real-time settlement in 2023, most credit unions will have to support them to satisfy members, Vuylsteke says, so they will need to plan for reliable connections.
The loss of delayed settlement will make it harder to unwind mistaken or fraudulent payments, Sharma points out. “Those systems will require FIs to have strong up-front processes,” he says.
Hackers up the stakes. It may be possible to create fast, complex payment systems that support multiple endpoints safely during peaceful operations, but now hostile attacks are exposing weaknesses in systems long considered safe. IT management software provider Kaseya (kaseya.com), which reportedly has CU clients, experienced a brutal ransomware attack over the July 4 holiday.
Hostile attacks underscore the need for protecting systems, Bain agrees, but he notes that those attacks usually don’t cause outages. “Criminals typically want to penetrate systems but keep them running while they do their dirty work.”
In all, the sky is not falling. “Of all the computer networks running today, across all industries,” Crone says, “the payments system is by far the most protected, but it is also the most relentlessly attacked because of its value.”
So far, it’s holding up well, he notes, especially the parts managed by financial institutions. The problems have come at the edges of the networks, he points out, like the merchant point-of-sale systems for card acceptance (e.g., McDonald’s) and among payment parties that are not financial institutions, but everyone is working hard to address those vulnerabilities.
© 2021 CU MANAGEMENT. REPRINTED WITH PERMISSION.
