A bank, many years ago, with a mainframe doing what it does best: process files and records with relentless steadiness.
For most of my life, I’ve been unofficial on call in some form or another. That evening was no exception: I received a call saying we had a prio 1 incident.
Upon arrival in the control room, this was the situation: one of the mainframes handling payment was apparently stuck in a processing loop. Lots of processing power was being expended, with millions of files being generated and database records being updated. All out of control.
Before I dive into the incident itself, here is the minimum you need to know about the context. This mainframe processed hundreds of thousands of transactions in flight with a hard commercial deadline the next morning. If we did not make it, clients would see a slipped value date and potentially fines would follow.
In that time, financial transactions were typically not processed in real time, unless you specifically asked for that service. They were handled overnight with something called batch processing. That is where you select all the transactions for the day, typically until 16:00, and start the process of getting the payment from account a to account b.
When that transaction stays within the same bank, that would be finished within that same evening, for a SEPA transaction, it could take another 24-48 hours and for SWIFT transaction, a few days. Today this is different, with most Sepa happening instantaneously within the EU. Ledger posting, typically the next day.
It was during this batch processing that things went wrong.
From a computer perspective we faced a simple initial problem: with millions of files being generated, we ran the risk of running out of disk space. Mainframes are very robust systems with operating systems that are rock solid, but running out of disk space is a very unhealthy situation.
At that moment, in incident management, you’re in the containment phase. You need to stop the bleeding. Your first instinct is thus to turn the thing off: we risk running out of space, more files are being created every second: stop creating files. The operators had not succeeded in halting the offending programs, thus… kill the mainframe.
An analogy is a burst pipe. To stop the bleeding you turn off the water supply. Simple. Unless you are in a nuclear plant, then you may want to know your rerouting options.
Mainframes, especially bare metal machines, really don’t like having their plug pulled like that. It is a very bad idea and would most likely make the situation much worse.
What did we do? Even though we would run out of space, that would not happen in the next 5 minutes. Like in most companies, we over-dimensioned the system to allow a lot of breathing room. Meaning that we would never use more than 45% of CPU, 50% of peak-use disk space (like the space you use during month-end or year-end processing) and so on.
That meant we could take a look at the larger picture and have time to bring in the real application and second-line systems experts. We have on-call rosters for incidents and a defined process for when to call up these people.
In parallel, we monitored the neighboring systems to determine where in the processing chain we had the problem. While some internal systems were processing some of the files, nothing was reaching any interbank related systems. That “limited” the footprint of the incident and helped calm nerves a bit.
We were together inside of 15 minutes in a virtual call. Yes, that is a very long time for a mainframe that is looping, but we were monitoring the situation carefully.
Once the experts had a chance to look at the situation, it became clear that the offending app could not simply be “abended”, mainframe speak for stopped. In fact, it should have abended by itself, but something prevented the program from breaking out of that logic loop. That would need to be looked at in the postmortem.
That brought us to the next step in incident management: mitigation. If I cannot stop the bleeding for that one system, what else can I do? Well, reroute. That is also not without danger and may place the bank in a situation where there is collateral damage.
Why is that? By rerouting, you can read that as “activate the standby system and isolate the sick system.” We know which program the issue is on, but we’re not sure what has already been updated in the databases. That means that the standby system will start with a dataset whose status is unknown.
We can’t have that. Hence, we need to restart the entire evening batch processing, but now we run the risk of losing a day in payment processing, incurring fines. Thus, once that was clear I escalated to the top IT management as we needed their input.
Luckily, our top IT management were people with relevant technical skills, but not everyone will be in that comfortable situation.
We took another 15 minutes to decide to activate the DR scenario for that system, once the application team had a chance to look at the standby system, the input files of that night and the set of pre-selected records in the database for processing.
An issue was not immediately found in the base input files, so we took the decision to recreate the files from the required databases and start afresh on the standby system.
Once the controlled DR happened and the secondary system took over in a semi-manual way (to avoid a repeat runaway process), we let the primary system exhaust its disk space and let the OS safeguards kick in, to bring the mainframe to a halt in a controlled way.
In the end, some payments did receive a value-date of +1, but that was the extent of it for the client. And that is the moral of the story. It shows the principle behind the DORA legislation. DORA does not tell you that you cannot have incidents. Obviously, the less the better, but you need to make the incident irrelevant to the client. How you clean it up internally is your business; just make sure the service works for your clients.
That leaves only one last item. Your audit trail. During the incident, keep running notes.
Like most war rooms, we used whiteboards to express our thinking: we write down the symptoms we see, we add a column for theories, actions, PTAs (persons to act) and results. We update and refine as we go along. For the records, I typically take pictures and add them into the incident records or at least keep them centrally. Those, combined with the eCAB (emergency change advisory board) held inside the war room with the relevant people present or on the call, complete your audit proof.
The incident is solved in the war room via clear judgement calls; the inspection is won in the records. In the heat of the moment, judgement calls are almost never recorded, and yet they will show the auditor that you are in control.