What's the Worst Case Scenario? Change Management and Predicting Disasters.
A useful change management process includes a documented worst case scenario as well as the disaster recovery plans for when that scenario takes place.
I'd bought a sailboat with a group of friends (other Amazon employees). We had been discussing signing up for sailing lessons together. I had shopped around, and found how much it would cost for the group to get lessons. Coincidentally, it was the same price as a lovely 23-foot sailboat. We decided that learning to sail on our boat was a better investment.
Fast-forward a few months, and our sailing skills were not terrible. We took many trips through Lake Union and Lake Washington, and felt competent enough to relax.
One day, we were out on Lake Union, and the breeze was really picking up. Our boat had a pretty heavy keel, which meant we were able to maintain sail while other boats were dropping theirs. We were cruising at high speeds, tipping further each crossing as the wind grew in strength.
While laughing and having a great time, one of the guys asked, "I know this feels a little scary, but what's the worst-case scenario here?"
We're all familiar with the change management process at Amazon, which requires defining the worst-case scenario.
I looked around, and then responded, "Well, the sail gets stuck up. And then the wind picks up more, and we're totally hosed."
He laughed, and agreed that would be the worst-case scenario.
The wind picked up.
We went to lower the sail, and nothing happened. It stayed up.
Upon inspection, a loose line had wrapped around the top of the mast, completely stopping our sail from being lowered.
This is 100% a true story. I mean, the level of coincidence is astonishing.
Over the next couple of hours, we sailed frantically back and forth across the lake, repeatedly coming to the brink of swamping.
Everyone tightened their life jackets. The water was cold, so we were concerned about making it to land if we tipped. We strongly considered ramming our boat full speed onto the shore. We discussed the merits of someone climbing the mast to untangle the cord, as we were all rock climbers. This seemed both possible, and highly dangerous.
Then, by a stroke of luck, we were able to unwrap the line, and the sail came down. Crisis averted. Heart rates returned to normal.
Change Management Process
When making a change to a company system, there is often some type of documentation regarding the change. Even if you're not at a company which enforces this type of strict process, going through the motions is useful.
The documentation includes obvious things like what is changing, how is it changing, when it is changing, and who is doing the changing.
One of the most critical parts of the change management process is analyzing how bad things can get. What is the worst case scenario, and what will we do if that worst case scenario happens?
What's a Worst Case Scenario?
Part of Amazon's change management (CM) process is documenting the worst case scenario. Far too often this step is not respected. I've read dozens of CMs which suggest that the worst case scenario of an update is "my change won't go live." In other CMs, the worst case scenario is "Customers will see errors."
I've also seen dozens of Sev-2/1 events (high severity events - people get called in the middle of the night - it's a big deal) where something worse than the documented Worst Case Scenario happened. What's worse - when you're unprepared for a worst case scenario, your ability to react is inhibited. Your potential 5 minute event became a 2 hour event, because you didn't think ahead.