I’ve been talking about code smells for a while, but code is far from the only thing that can go wrong. What about process smells? There are things in your development process that may not necessarily be wrong in every situation, but are a signal that something could be wrong. Some of these are more definitely stinky than code smells, but not all of them. As always, if you’re happy and productive with the way your process works, stick with it!
Here’s an example of a process smell: Friday prouction deploys (in case that tweet ever gets deleted or something, it’s a picture of a jack o’lantern with the words “Friday deploy to prod” carved into it and the caption “Scariest Pumpkin”). Friday deploys are almost universally a terrible idea, which makes them an especially nasty process smell, but once in a while they are actually a reasonable choice. Usually when they’re the least terrible of a set of bad options, but sometimes bad options are all you have.
If you’re fixing a severe bug that prevents a lot people from using the system at all, then it makes sense to deploy whenever you have a fix, even if that’s a Friday, even if it’s late in the day on a Friday. And if you have the unusual luck to have a dev team with staggered schedules who are all familiar with the code being deployed, then no day is really any riskier than any other (is this actually a thing? I included it because it’s technically possible but I’m not sure it actually happens anywhere).
If you work in a very rigid environment where nobody uses the system over the weekend at all ever, then knock yourself out! If no one touches the system over the weekend, then you’re not going to hurt anything by deploying on a Friday.
Most of the time Friday deploys are just a bad idea. Why?
Because no matter how great your QA department is, even they can’t test every single possible combination of weird commands and bad data and dodgy wifi that your actual users or customers can, which means things may break in wildly bizarre ways when no one is around to fix them. That sucks for your customers and sucks for your support team and sucks for your revenue and sucks for your reputation and sucks for your stress levels when you come back and discover that everything has been on fire all weekend. Oh and it sucks if you get called in to fix it when you thought you were going to have a nice relaxing weekend. Spare yourself and the rest of your team a lot of hassle and don’t deploy on a Friday (or late afternoon any day) if you can avoid it.
This process smell is really interesting if you keep in mind this definition of a code smell, by Martin Fowler:
A code smell is a surface indication that usually corresponds to a deeper problem in the system.
If you’re deploying to production on Fridays, there’s definitely a deeper problem. Who is even letting you do Friday deploys when they’re so widely acknowledged to be a bad idea? Or who is pressuring you to do them when you know they’re a bad idea? And why is that person getting away with it? Who is pressuring them? Why is it more important to say something went out than to be sure that somebody is around to fix it if anything breaks? Or, and this is really scary, is management assuming that developers can be called in any time night or day to fix things?
Unfortunately, there isn’t a neat set of fixes for process smells the way there is for code smells. If I could give you a few quick and easy steps to convincing your company that deploying to production on a Friday is a bad idea, I’d be a millionaire :) What I can tell you is that you need to understand why your team lead / manager / upper management thinks it’s a good idea to deploy anything on a Friday before you can convince them to change. If the problem is that upper management believes it’s lazy to wait until Monday when the deploy could be done on Friday, then explaining that deploying on Fridays hurts team morale by making people worry their weekend plans will be wrecked by a surprise issue isn’t going to convince them anything needs to change. On the other hand, if you remind them how angry your customers were the last time production went down on a weekend and stayed down for hours because that particular release wasn’t possible to quickly roll back and the developer who knew the most about that change was away on a long-awaited hiking trip, and tell them there’s a way to prevent that from happening again that costs zero dollars, you’ve got a much stronger case.
2 Comments
Bob Warwick
Devils advocate: There’s been plenty of software written with processes that don’t allow for immediate hot fixing. Boxed software, software that requires third party app store review for distribution, etc. We still ship good software, despite not being able to come in and fix it when we get the first bug report. Our bar for what constitutes a QA pass is pretty high.
Is it a process smell that you’re relying on the ability to instantly deploy? How would the software you deploy and your process change if you weren’t able to rely on hot fixes? Are you outsourcing your QA to customers? Would it be a net benefit to have higher quality standards so that shipping on Friday isn’t scary, or does the time and cost make that less than worthwhile?
(I’m honestly not sure. I believe QA, both automated and human has to be part of a software release process, but I also fully acknowledge that you can’t anticipate every issue. The ability to hot fix is fantastic, and it moves you from the version-number model of software release, but I’m not sold that that’s a win for your customer.)
Mel Reams
You make a good point. A few years ago a local game studio got rave reviews for their game because they happened to have released it after a year of buggy AAA-studio releases and people were really excited about getting to play a game that actually worked. Even with the ability to provide updates for console games, people were pissed they couldn’t play the games they paid for right away.
Would it be a net benefit to have higher quality standards so that shipping on Friday isn’t scary, or does the time and cost make that less than worthwhile?
I don’t know either, but that’s a really interesting question. There are certainly some features I’ve released that would have benefited from more testing, and if I were to go back and ask the customers who found those bugs if they would rather we had waited to release those features at all I’m not sure what they’d say.
There’s this belief that you have to be first out of the gate with new features or your customers will all ditch you for the competition, but I’m not sure that’s actually true – at least not for B2B products. For something like free-to-play games then yeah, you probably do need to grind features out as fast as you can, but once you’ve invested in researching and integrating a product then you’re not going to drop it instantly.
I think not having to wait months for updates is generally a win, but I also think we’re overcorrected as an industry to the point where we do outsource QA to our customers at least some of the time. It probably would be better for customers to have not-quite-as-fast updates that work more reliably, although with the sheer volume of users software has to handle these days I have to admit it’s pretty hard to catch every possible bug.