Prepare for the Best Case

I’m in terrible shape right now, and I’m trying to get back to where I was maybe three years ago. The goal is to run a Spartan race sometime next year.

I was never a good athlete, but I did run my share of races: Spartans, 10Ks, half-marathons, full marathons, and even triathlons. Most were small events, but the marathons I ran at Disney World were huge.

The Walt Disney World Marathon Weekend includes four races and attracts about 30,000 people who show up at the same time and need to register, sign waivers, and pick up their running kits.

That takes a lot of logistics and organization.

I’ll tell you where I’m going with that in a second, but here is another one.

When I got my job at Google, the onboarding was at headquarters in Sunnyvale. More than 300 people started in the same cohort as me. Nevertheless, it was the smoothest onboarding of my entire career to this day.

When I got on-site, the signs clearly showed me where to go. The security guard at the door checked my documents, and as I was let in, I joined a line. I don’t remember the details, as it was more than 10 years ago. Still, I remember handing over some signed paperwork I brought with me, receiving a laptop I was instructed to log in to right away, which unlocked access to other systems that, in a wizard-like way, walked me through everything.

Despite hundreds of people starting, between entering the building and being fully onboarded with my own laptop, credentials, 2FA, bag, swag, and sitting in the huge training room, not even 30 minutes had passed.

Another huge logistics and organizational accomplishment.

I am sure many people have experience with this kind of planned event, when there is a huge influx of work and somehow it gets managed beautifully.

But the key word here is “planned.”

Unplanned events are much harder to handle. They require an emergency plan, constant drills, and time and resources spent preparing for something that may not happen. A hard sell.

Smart companies call it DR (Disaster Recovery), as the idea is that if something terrible happens, they can quickly restore their systems and continue operating.

This is excellent and very underrated. Many more companies should be preparing for a worst-case scenario. I wrote about one piece of that in Restore or It Didn’t Happen: a backup you have never restored is only a hypothesis.

But not many companies prepare for something else: the best-case scenario.

I posted this week on LinkedIn about a situation that happened in Paraguay. Early this week, the Central Bank shut down Ueno, one of the country’s banks, for insolvency. Accounts were frozen. The government moved fast on refunds: if you already had an account at another bank, they would find it and credit your money there.

The plan sounded reasonable. But about four million people were affected, and many of them suddenly needed a new bank account at the same time. Pretty much every bank in the country was either down or struggling. I read that Itaú, the largest bank in Paraguay, usually processes around 10,000 account applications a month. The day before the refund plan was even announced, they handled 12,000.

Nobody at those banks pushed a bad deploy. Their systems were sized for a normal month. A normal month did not show up.

What showed up was a surge caused by someone else’s failure. A policy decision turned into a load test across an entire industry.

That is the best-case scenario nobody plans for. Your product goes viral. A competitor disappears overnight. A government program dumps a million new users on you. Your sales team lands the deal of the decade. From the outside it looks like success. From the inside it looks like every dashboard lighting up at once.

A few things keep coming back to me.

Capacity planning has to include “what if a competitor disappears overnight?” Your worst day might be caused by someone else’s bad day.

Watch the golden signals. Latency, traffic, errors, and saturation would have shown this coming hours before customers felt it. Traffic jumped before the announcement.

Elastic infrastructure helps, but only if the whole chain scales. You can add servers in minutes. You cannot add KYC reviewers, call-center staff, or downstream partners that fast. The bottleneck moves to the human step, and that is where the queue forms.

Degrade gracefully. A page that says “we got your request, expect an email in 48 hours” beats a login screen that times out. Planned events like Disney and Google work because they turn a flood into a line. Unplanned floods need the same trick: absorb the surge, then process it.

Back to the Spartan race.

Getting ready for a race I might run next year means training for a day that is still optional: spending money and sweat on something that may not happen. That is the same hard sell as DR, and the same hard sell as capacity for a surge you hope never arrives.

The companies that handle both the bad day and the good day are the ones that practiced when nothing was on fire.

A policy decision can turn into a load test across a whole industry. The question is whether you ran that test first, or your customers did.

If a good week of growth would break your systems the same way an outage would, my two-week Systems Map finds the bottlenecks before your customers do. Here’s what I offer.