3 DevOps practices to avoid the Thursday Night of Dread

Recently, I described DevOps as an investment in team happiness. It can mean actually looking forward to deployments, rather than fearing the Thursday Night of Dread. In particular, I’ve found three specific DevOps practices that go the farthest in boosting team morale:

Fast feedback through automation and telemetry

You have to commit—really commit—to automation and telemetry. People sometimes claim they’re doing test-driven development when they’re actually relying on manual testing. They say automation is too hard, or they can just get an army of offshore QA folks to do the manual testing. 

But that’s not good enough: you need feedback in milliseconds—not hours, certainly not days. Manual QA, performed by armies or anyone else, will never be fast enough. It will always create a bottleneck. If every release takes four weeks or more of manual regression, that’s as fast as your development cycle will ever run. 

Over the long term, it's always cheaper to automate. It’s also much easier on your team, and frees up their headspace to do the innovative work you hired them for. 

Frictionless flow through streamlined procedures

The point of DevOps is to move from idea to code to deployment as quickly as possible. To do that, you need to ruthlessly eliminate friction wherever you find it.

Hand-offs are a common bottleneck, especially when you’re handing off across time-zones. Executives like to boast about development teams that “follow the sun.” In practice, following the sun often means injecting a day of delay into every work cycle. One team identifies a problem and goes to sleep. The next team fixes the problem and pushes the code for review. Twelve hours later, the first team wakes up to find the code still not running properly, and the cycle repeats. Instead of following the sun, you end up chasing the sun.

Sometimes you get the most ROI from streamlining your workflow: pruning useless activities, getting rid of non-value-adding policies that add work without adding value. In high-performing organizations, I've heard leaders tell staff: "Don't let the process get in the way. Have a conversation. Do what's right.” 

In others, I’ve seen team leads force staff to execute outdated, meaningless procedures just to check a box. Guess which had better was more productive?

Repeatable rehearsals build muscle memory

You should rehearse production release and monitoring as part of your daily work. When it comes time to deploy, you want the team to be at ease. They should be confident that they’ll know right away that the rollout is either going to work flawlessly, or, if something goes wrong, the rollback will be so fast and painless that failure will be nothing to worry about. 

You get to that point by deploying the same way multiple times a day, every day, for months. Store everything securely, under source control (app configuration, server settings, network config, etc.). Automate the entire deployment process, and have great telemetry from your infrastructure and application. When you have all that in place, you can deploy by running a command or pushing a button, and the machine you’ve been practicing with kicks into gear. If something goes wrong, you activate the automated rollback. It should all get done in minutes.

You may have noticed, what binds all these practices together is the need for discipline. Doing DevOps means choosing a slightly harder course now for a much easier time later. But like exercise, I’ve never found anyone who regretted doing it once it’s done.

As published in InfoWorld