The Drive of Shame

There was a time when I was younger, doing some network maintenance for a client whose ISP was providing a fibre link between two locations. We were only an intermediary, since a large telecom operated the dark fibre, but our role included network setup and maintenance at both sites.

My memory has faded over time, so I don’t remember exactly what the company did, but they were in agribusiness. Their headquarters were in São Paulo, where I was located, and the other site was a distribution center in a rural part of the state, about 200 km away.

We were setting up some new internal switches in São Paulo, replacing older models with something faster and with more ports. There were VLAN configurations, trunks, encapsulations, and all that fun stuff that had to be put in place. I had two terminal sessions open and kept flipping between them to get things done.

The most experienced of you already know where this is going.

Of course, at some point I messed up and entered a shut command on the wrong switch.

The one 200 km away.

That client had no in-house IT staff. The distribution center was fully populated with blue-collar workers who were absolutely allergic to technology. I called, but they didn’t even know what a switch was, let alone where it might be located.

So I had to do the drive of shame.

But that was the last drive of shame in my career.

After that, it was only the taxi of shame, many walks of shame, and plenty of calls of shame.

Like most people who have spent enough time working with technology, I’ve made my share of mistakes. Some were simple, like shutting down the wrong switch port and noticing immediately. Others happened because I didn’t fully understand how a system would behave under a particular set of circumstances.

Here’s one example that, to this day, I still hold a grudge against Microsoft for.

I worked for a SaaS-based PoS company. Everything was on-premises and, at that point, virtualized using VMware.

There was proper separation between the internal network and customer-facing servers, with the expected firewall layers, DMZ setup, and all that jazz.

We did, however, have one piece of shared infrastructure: the Windows domain controllers also served as the authoritative DNS and DHCP servers for both the internal and customer-facing environments. We had several subnets, including one used for development and testing, where I would often spin up servers to experiment with things.

This setup allowed internal staff to access application servers through internal names and internal routing instead of sending traffic out to the internet and back again.

That said, customers frequently broke things in their configurations or databases, and simply restoring a backup was rarely a good option. More often than not, I had to spin up a full snapshot of a customer’s environment so that support or engineering could log in and selectively recover data.

Around that time, we had adopted a new backup solution with native VMware integration. It was fantastic. I could run a virtual machine directly from a backup snapshot in less than ten minutes.

So one evening, when a customer request came in late in the day, I spun up a copy of the shared server they were hosted on in our development subnet and went home.

The next morning, around 7 AM, the on-call technician called me in full panic mode. One of our early-rising customers was completely unable to access their PoS system.

He had already done some troubleshooting and couldn’t figure out what was wrong. The server looked healthy. Other customers on the same host were happily chugging along. Nothing obvious stood out.

It took me a while to figure out what had happened.

The Windows domain controller provided a DHCP address to the server running from the backup snapshot. That server then announced itself using the same hostname as the production server. Dynamic DNS happily accepted the registration, causing some internal systems to redirect traffic there.

And since that machine lived on a different subnet with different firewall rules those connections failed.

The result was that this particular customer couldn’t connect.

Everyone else was unaffected because they left their PoS systems running overnight. Their sessions remained established and continued working normally. This one store happened to shut everything down every evening and start fresh every morning, making them the only customer exposed to the problem.

At the end of the day, I still take responsibility for that outage.

Whether I agreed with Microsoft’s behaviour or not didn’t matter.

It was my change, my environment, and my responsibility to understand the consequences before I walked away.

Throughout my career, I’ve seen people make all kinds of mistakes and cause all sorts of outages and pain.

Yet I’ve never seen someone permanently derail their career because of an honest mistake.

People mess up.

It’s part of working with complex systems.

What ruins careers is trying to hide the mistake.

The moment you realize something is wrong, your only job is to stop thinking about your ego and start thinking about the customer.

Raise your hand.

Tell your manager.

Call the senior engineer.

Wake someone up if you have to.

The sooner more eyes are on the problem, the sooner the damage stops growing.

I’ve seen engineers spend thirty minutes trying to “fix it before anyone notices.”

Those thirty minutes are often the most expensive part of the incident.

While they’re trying to protect their reputation, customers are still down, logs are being overwritten, evidence is disappearing, and the blast radius keeps expanding.

Ironically, admitting the mistake immediately usually has the opposite effect.

When someone says, “I did this. I don’t know how to fix it. I need help,” my confidence in that person goes up, not down.

They’re demonstrating ownership.

They’re putting the customer ahead of their pride.

They’re optimizing for recovery instead of appearances.

Every experienced engineer has their own collection of drives of shame, walks of shame, emergency phone calls, and outages they’ll never forget.

They’re almost a rite of passage.

The difference between a junior engineer and a senior one isn’t that the senior engineer never makes mistakes.

It’s that they recognize them quickly, communicate them immediately, and bring the right people into the room before a small problem becomes a catastrophe.