Everybody screws up sometimes and I wish we talked about it more. One of many things that I think scares off new programmers is the belief that if you ever fuck up then you’re just not cut out to be a programmer. That’s totally not true, even experienced professional programmers make mistakes. All making a mistake means is that you tried to do something that’s actually difficult, it’s not the end of the world.
In the spirit of helping other people not freak out about their own mistakes, let’s talk about some of mine.
Early in my career I came up with a truly terrible design for a portlet (remember portlets? No? Okay, I feel old now). If I remember correctly it was supposed to provide a friendly UI for people to create their own report queries, so there were all sorts of rules about which kinds of values applied to each field and how fields could be combined to keep track of. This was back before I really understood that requirements never ever stay the same, and after a couple rounds of changes my design was a real mess. If nothing had ever changed my house of cards might have been okay, but since things did it started to crack and warp under the strain.
What did I learn from that one? That requirements always change and your design needs to handle that, and that it’s always a good idea to have someone else review your design before you start building, especially if you don’t have a lot of experience. Designing software to handle requirement changes is something I’m still working on and something that could fill a whole book, let alone a blog post, so I’ll just say that asking yourself what happens if you need to add more rules or take some away is a good start.
Another thing I messed up I already mentioned in my post If it’s hard to explain, it’s probably a bad idea. To recap really quickly, we (the backend team on the project) came up with a scheme to sort players into rooms that was basically impossible to explain to non-programmers. Actually, I’m not sure it ever even made sense to the devs on the client team either. I bring that one up again to drive home the point that design is hard and you’re going to screw it up a lot before you get good at it.
What I learned from that mistake was that designing in isolation can really trip you up. That design seemed like a good idea to us backend devs in isolation, but if we had run it past the rest of the team first we could have figured out it was confusing and a bad experience for the users before we wasted weeks building something that ended up being confusing and frustrating for the people who actually used it.
I’ve also really fouled things up with databases. I once took down production by making a data model change that had run fine in test. Turns out things run differently when a) there’s actual load on the database, and b) there are orders of magnitude more rows in prod than in test. Now I know not to try to make data model changes in mysql cluster without taking the app down for maintenance.
Sadly, that’s not the only time a difference between prod and test has tripped me up. The amount of data in your database can change the way mysql optimizes its queries, which can lead to some really interesting results when you assume the database is using an index that it’s actually blissfully ignoring. And by interesting I mean unusably slow response times. Wheee.
What that one really drove home for me is that you can’t trust the test database unless there’s a full production dump in there and you’re simulating realistic load. Outside of a perfect clone of production, load included, you need to lean on explain plans. Explain plans are great because they tell you how things are going to run into production without actually migrating your whole application to prod and breaking everything. Next time, I’m running explain plans first in production instead of assuming things will be fine because it worked in a test database with a few hundred rows.
Another really entertaining screw up I’ve had was when we were preparing to turn off an online game that wasn’t performing well and was a maintenance nightmare on top of that. We thought that it would be a nice gesture to make the virtual game cards (kind of like a virtual scratch ticket) in the game free so they could play as much as they wanted until we turned the game off entirely. The design of the game didn’t allow us to make the cards entirely free, but it did allow us to make everything cost only one token. Normally higher level game cards cost hundreds of tokens, so we figured players were still getting a pretty good deal.
Unfortunately, I didn’t think through the consequences of players being able to buy hundreds more game cards than usual. On each card you could win credits (thankfully you couldn’t buy more cards with those credits or we really would have been in trouble) that you could use to buy prizes in the store. Normally the number of credits you could possibly win at one time was limited by how many cards you could afford, with the higher level ones that gave out lots of credits being so expensive that you could only buy a few at a time.
Does anyone see where this is going? Yes, overflows! We let players win so many credits that the fields in the database couldn’t hold their winnings. Once we fixed the database we discovered that parts of the game server code couldn’t handle those numbers either. It was a rough few days trying to get that game back into action.
There are two main things I learned from that experience, one specific to the problems we had and one less so.
One – do not make sweeping changes to a game’s economy! Just don’t do it, it’s a bad idea and will make you unhappy. The next time we sunsetted a game (the industry’s polite euphemism for killing it) we most certainly did not destroy the economy on the way out.
Two – the consequences of an action aren’t always obvious, you’ve got to put some work into figuring out what’s going to happen when you change things. If I had that particular change to do over again, I would look at how many daily tokens our top players were getting and calculate how many game cards they could buy with those tokens and how many credits they could win off of those game cards. Then I would try manually giving myself that many credits and tokens in test and see what happens. After watching everything blow up, I would go back to the product manager and say “1 credit game cards break the game, how about we just give players 50% off?”
Let’s mix things up a little. I have one more failure to talk about, but this time it’s not one of mine. Here’s the notification I got about a sad little troll trying to comment on my Shitty hackathon! post:
New comment on your post “Shitty hackathon!”
Date: 10 February, 2016 11:43
Author: Nancy Grace (IP: 188.8.131.52, d72-39-146-43.home1.cgocable.net)
Comment: I must say, that I am impressed. Keep it up, and one day you just might earn your official script kiddie participation award. In the meantime, don’t quit your night prostitution job just yet ;)
Now, it’s pretty clear that this shitheap is unable to learn (if he was, he would be good enough at his job not to be frightened of a woman talking about code), but other readers can learn something from this. First of all, it’s pretty clear that he’s terrified of women developers. That in turn makes it obvious he sucks at his job, because with the shortage of developers you have to be a serious fuckup to be afraid a girl will come and steal your job. It’s also pretty sad that he had to go digging three pages back from my front page. Poor bastard had to go past two of my posts about SOLID design principles and my post about Java synchronization and my post about different languages being good for different things before he found something that he thought he could smear his poop on like the shitty little baby he is.
If you’re going to try to shit on me for not being a perfect programmer, get it right! I’m embarrassed for you and I don’t even like you, fuckstick.
However, fuckstick does accidentally bring up something in the vague (astronomically vague) vicinity of a point: if you talk openly about your failures, it’s possible someone will give you shit for having the unbelievable gall to be a human instead of a robot. You should ignore those people because they’re stupid :) No seriously, you do not want to work with or for anyone who is so willfully ignorant of what software development actually is that they think it’s possible to do this without ever making mistakes. If god forbid your boss shows you that they know that little about development, they’ve given you the gift of complete certainty that you need to run. Everybody in Victoria is hiring right now and developers are in demand all over, there’s no reason to put up with a boss who doesn’t understand their industry.
Another reason to talk openly about your failures, especially on your blog, is that when the classic interview question comes up, you’ll have something to say already lined up. You could even share a link to your blog post if you’re less of a potty mouth than I am :)
How about you, readers? What have you screwed up and what did you learn?