DevOps: cure for the Thursday Night of Dread

Whenever someone asks me to explain DevOps, I always start with the same story. It isn’t the usual story about time and money. It’s about people.

I used to work on two web applications at a large healthcare organization. Technically speaking, there wasn’t a whole lot of difference between them: back-end services, a couple supporting Windows microservices, same data center. Pretty standard.

But on deployment Thursdays, the two couldn’t have been more different. 

One side of the house had invested heavily in automation. They had everything under source control. The entire deployment process was planned, scripted and logged.Our weekly deployment calls took a quick five minutes to sign off.

On the other side, deployment was done manually with lots of different people on different teams, in different time zones. This resulted in a very different experience. 

I called it The Thursday Night of Dread

See if this sounds familiar. You race home at 5:00 so you have time to eat dinner and get the kids settled before the conference call at 10 pm.  You grab something caffeinated in case things really go south and dial in. It would have been 9:00, but the newly minted business VP asked it to be moved back an hour so that clients in Hawaii can log out of the system (you don't have clients in Hawaii). 

Ten minutes in, everyone’s on except the DBA, who's hung up on another line dealing with a Sev 1 outage. At 10:20 the business guy who made you move it back to 10:00 says to text him if he's needed and drops off the call. It’s 10:23 before the DBA joins the call and you can get started. You slog through the deployment plan, one manual step at a time: manual backup, manual config, manual deploy -- each group executing its steps and updating everyone on the call. 

It’s past midnight before you finally get everything up and running. QA swings into action and quickly discovers a service that isn’t restarting correctly. You split off a smaller team to triage the issue. After 30 minutes, the verdict is in: the service is toast. You have to roll back. 

Now the real dread sets in as you down your stand-by coffee and begin the painful process of undoing the night’s work.  It’s 4:30 in the morning before you agree to get some sleep and triage the issue the next day.  Eventually someone traces the problem due to an undocumented manual configuration change: a ten-minute fix if you’d known.  Instead, everyone loses a night’s sleep—and gets to do it all over again next week.

This sad scenario is a weekly reality for countless unlucky developers, administrators and quality engineers. It’s bad for them and their families. It’s also bad for business – and completely avoidable.

The benefits of DevOps are well known: faster feature delivery, more efficient use of resources, better stability. But the best reason is eliminating the Thursday Night of Dread. 

At a time when every company worries about closing the innovation gap, the last thing you want is to demoralize your staff and sap their creativity. The cost of burnout is real. 

Writing deployment scripts always sounds like a good project for next quarter. Coding telemetry seems like such a chore. But it’s nothing compared to the pain of a failed four-coffee, eight-hour deployment. 

Done right, DevOps is more than a tool for efficiency. It’s an investment in employee happiness. 

In my next post, I’ll offer up some strategies for accelerating your DevOps return on investment.